Oddly, AFAICT no one has appealed to the First Vatican Council yet. If we consider the dogmatic declarations of ecumenical councils to be infallible, then you have your evidence, because an ecumenical council defined this as dogma
The Council devoted an entire chapter of its
decrees to the subject. The main thrust of their reasoning is (1) that Peter, and Peter alone, was entrusted with the ministry of confirming the brethren, and (2) it was fitting that this should occur, since without a central touchstone of truth, schism comes easily (for if two men of equal teaching authority disagree, how do you know which one to believe?). There are other arguments in that chapter, though, including some quotes from other relevant councils.
This, of course, depends on the idea that all dogmatic definitions of ecumenical councils are true. But that claim has its own Scriptural support (Mt 18:18, Lk 10:16), and rejecting the infallibility of the councils gets you (in my opinion) perilously close to sola Scriptura.
I wish Vatican I had put in more citations in its decrees, but I hope the text helps a little bit.