When did the apostles say this? According to the Eastern Rites, Rome does not have infallibility, and they actually have valid apostolic succession as well dating back to apostles who were also promised the same authority as Peter.
The Orthodox do, however, recognize that bishops are successors of the apostles, even if they do not think that Peter had the same power that we think he had. So far as I understand, they do not deny that the Pope is the successor of Peter, but simply deny that Peter’s chair has the authority we say it has. This is a separate (though highly related) issue though, so I’d just as soon leave it for now. After all, it makes no sense to talk about a pope if we don’t have bishops.
Going back to the other issue, I’ll admit that there’s a little bit of trust involved here. Not an awful lot - Acts clearly shows the apostles choosing successors and the mere word “successor” implies continuation of the same function - but some.
The best I can say on short notice without digging through sources (which I just don’t have time to do right now) is that, despite what protestants tend to say, the early Church Fathers clearly have the same opinion of bishops as we do today, and these people would only have been a few “generations” of bishops removed from the Apostles themselves.
So the only way this could be wrong is if the apostles ordained successors and
did not tell them that the infallibility given to the apostles as a whole (and to Peter in a special way, but that’s a different, though related, story) was passed on along, and those successors ordained successors and did not tell them that they had, as a group, the charism of infallibility, and so on, but within the first several “generations” the entire collected body of bishops (or enough that it might as well have been the entire collected body) claimed to have this power for no apparent reason
WITHOUT CAUSING A BLOODY ROW.
Now, the closer in time you are to the original apostles, the harder it would be to mess with doctrine because you’d have people saying “I remember Peter or Paul saying that you’re wrong.” But the farther you are from the apostles, the harder it would be to convince all the bishops at once that they had some sort of power that they had no reason to think they had, simply because there’d be more of them.
So for this doctrine to have spontaneously generated would have required either a vast conspiracy or a vast stupidity among all those people who the Church thought were the best suited to lead them, and it would have required this to happen without anyone noticing. This is extremely unlikely.
It would also amount to vast heresy by all the Church leaders. This is where the gates of hell argument comes in. You may not accept that that argument directly implies that the Church would be infallible, but you have to admit that the entire church falling into heresy for at least 1000 years, and more likely 1400 or so years, would be an awful lot like the gates of hell prevailing for a little bit.
Several people have already given several good arguments about
why Christ would have given the Church infallibility (after all, if a good shepherd never abandons a sheep, then clearly the Good Shepherd wouldn’t abandon His flock to figure out such things like the how to attain (or receive or whatever) salvation by taking their best guess at what a book means). But this is the best (on short notice) that I can do to demonstrate that Christ actually did, without recourse to the “it’d be best and Christ does what’s best” argument (though I like and accept that argument as well).