The portion of the Church that stayed with St. Peter is the one with the Authority of Christ.
Lu 22:31 “Simon, Simon, behold, Satan demanded to have you [plural - all the Apostles], that he might sift you [plural] like wheat,
32 but I have prayed for you [singular - Peter alone] that your [singular - literally “the faith of you”] faith may not fail; and when you have turned again, strengthen your brethren.”
This was at the last supper, right after He had promised them:
Lu 22:28 "You are those who have continued with me in my trials;
29 and I assign to you, as my Father assigned to me, a kingdom,
30 that you may eat and drink at my table in my kingdom, and sit on thrones judging the twelve tribes of Israel.
So, He promises them a kingdom, but that Satan wants to sift them all like wheat. Jesus, though has prayed for St. Peter alone, and wants St. Peter to strengthen the bretheren in the faith.