You want to restrict it to those who are in union with the Bishop of Rome. I don’t believe that is true from a biblical or early church perspective.
The Bible is very clear when it states that St. Peter was to be the leader of the Church and that his office would perpetuate after his death. Take Isaiah 22:22, in the old Davidic kingdom, there were royal ministers who conducted the liturgical worship and bound the people in teaching and doctrine. But there was also a Prime Minister or chief steward of the kingdom who held the keys. Jesus gives Peter these keys to His earthly kingdom, the Church. This representative has decision-making authority over the people - when he shuts, no one opens. The passage further states that the “prime minister” of Israel will fall astray, but his office (the keys) would remain, even if he fell into personal faults.
The Early Church made it unequivocally clear that the Episcopal See in Rome held precedence over all other sees:
“Since, however, it would be very tedious, in such a volume as this, to reckon up the successions of all the Churches, we do put to confusion all those who, in whatever manner, whether by an evil self-pleasing, by vainglory, or by blindness and perverse opinion, assemble in unauthorized meetings; [we do this, I say,] by indicating that tradition derived from the apostles, of the very great, the very ancient, and universally known Church founded and organized at Rome by the two most glorious apostles, Peter and Paul; as also [by pointing out] the faith preached to men, which comes down to our time by means of the successions of the bishops. For it is a matter of necessity that every Church should agree with this Church, on account of its pre- eminent authority, that is, the faithful everywhere, inasmuch as the apostolical tradition has been preserved continuously by those [faithful men] who exist everywhere.” **Irenaeus, Against Heresies, 3:3:2 (A.D. 180). **
“And he says to him again after the resurrection, ‘Feed my sheep.’ It is on him that he builds the Church, and to him that he entrusts the sheep to feed. And although he assigns a like power to all the apostles, yet he founded a single Chair, thus establishing by his own authority the source and hallmark of the (Church’s) oneness. No doubt the others were all that Peter was, but a primacy is given to Peter, and it is (thus) made clear that there is but one flock which is to be fed by all the apostles in common accord. If a man does not hold fast to this oneness of Peter, does he imagine that he still holds the faith? If he deserts the Chair of Peter upon whom the Church was built, has he still confidence that he is in the Church? This unity firmly should we hold and maintain, especially we bishops, presiding in the Church, in order that we may approve the episcopate itself to be the one and undivided.” **Cyprian, The Unity of the Church, 4-5 (A.D. 251-256). **
“After such things as these, moreover, they still dare–a false bishop having been appointed for them by, heretics–to set sail and to bear letters from schismatic and profane persons to the throne of Peter, and to the chief church whence priestly unity takes its source; and not to consider that these were the Romans whose faith was praised in the preaching of the apostle, to whom faithlessness could have no access.” **Cyprian, To Cornelius, Epistle 54/59:14 (A.D. 252). **
”The reason for your absence was both honorable and imperative, that the schismatic wolves might not rob and plunder by stealth nor the heretical dogs bark madly in the rapid fury nor the very serpent, the devil, discharge his blasphemous venom. So it seems to us right and altogether fitting that priests of the Lord from each and every province should report to their head, that is, to the See of Peter, the Apostle."** Council of Sardica, To Pope Julius (A.D. 342). **
"…I think it my duty to consult the chair of Peter, and to turn to a church whose faith has been praised by Paul…The fruitful soil of Rome, when it receives the pure seed of the Lord, bears fruit an hundredfold…My words are spoken to the successor of the fisherman, to the disciple of the cross. As I follow no leader save Christ, so I communicate with none but your blessedness, that is with the chair of Peter. For this, I know, is the rock on which the church is built! This is the house where alone the paschal lamb can be rightly eaten. This is the ark of Noah, and he who is not found in it shall perish when the flood prevails.” **Jerome, To Pope Damasus, Epistle 15:1-2 (A.D. 375). **
“Number the bishops from the See of Peter itself. And in that order of Fathers see who has succeeded whom. That is the rock against which the gates of hell do not prevail” **Augustine, Psalm against the Party of Donatus, 18 (A.D. 393). **
What I cited from Yves Congar, and the conclusions of Schaff, Kelly, and Pelikan in 
