Just a musing on your question: We currently have a very specific view of the papacy, one that the Eastern Churches seem to view a bit differently, and (obviously0 the Orthodox view a good bit differently. It would seem, from a review of 2000 years of history or thereabouts, that the pope was largely seen as the “court of last resort”, or the “final arbiter” over a large part of the history of the Church’ it is not that his charism of infallibility was questioned so much as that a lot of issues got hammered out without necessarily involving him, or necessarily involving him to a great degree. For example, there were several councils, at least, in which he did not attend, and they became “official” due to his later ratification. the current view seems to often see the pope as the head and leader of the church, with everyone else essentially doing his bidding. There have been discussions of papal infallibility which seem to count something like 20 or 30 instances of “ex cathedra” statements, but the consensus of most theologians limits those to two instances.
Thus it would seem that the question “what is the point of having infallibility at all” is one that sees infallibility resting primarily on the pope, as opposed to the Church through both it’s continued Magisterial teaching, and councils, and, finally, with the pope as necessitated when the other two for whatever reason don’t answer the issue. It might be remembered that the Church survived 1800+ years without defining his charism; he had the last say, but wasn’t the first to speak, as it were.
Or perhaps another way of saying it would be to look to the Orthodox, who often speak of the pope as the “first among equals”; while I don’t suggest that is the final answer, that may have a way of bringing some additional perspective to the issue.