You understand, of course, that this was part of Christ forgiving Peter for abandoning him.
Christ did not say ‘you alone’ feed my sheep. He was restoring Saint Peter and telling him to get off the beach feeling sorry for himself and back to work.
In scripture, Christ does not say “I will give to thee
alone the keys of the kingdom of heaven”. He says “I will give to thee the keys of the kingdom of heaven. And whatsoever thou shalt bind upon earth, it shall be bound also in heaven: and whatsoever thou shalt loose upon earth, it shall be loosed also in heaven”. Many keys, and he does
not say Peter alone is to receive them. Matt 16:19
Surely, if Jesus had intended Peter alone to have the keys, Matthew could have stated this right at this point, but he does not, something else is going on here, the narrative is not finished.
We have to deal with the literature as written. The power of the keys is the power to bind and loose (which the Papacy does try to monopolize at times) clearly stated by Christ in the same verse.
After Jesus acknowledged Peter’s faith and having promised Saint Peter the keys, Jesus Christ shocks us by declaring “Go behind me, Satan,
thou art a scandal unto me: because thou savourest not the things that are of God, but the things that are of men.” He is actually calling Peter Satan.

Matt 16:23
Later Jesus is speaking to
all of the Apostles. They come to him as this passage explains “At that hour the disciples came to Jesus, saying: Who thinkest thou is the greater in the kingdom of heaven?” Hmmm
Jesus launches into a long explanation of how the humility of a child is required of them all, and that they through their sin should not cause the loss of even one innocent person and they will be answerable for their souls. “your Father in heaven is not willing that any of these little ones should be lost.” He is telling them how to be bishops in this whole account which last several verses. Then he says to them all “Amen I say to you,
whatsoever you shall bind upon earth, shall be bound also in heaven; and whatsoever you shall loose upon earth, shall be loosed also in heaven.” The risen Lord has commanded them.
Later is this interesting remark by the Lord to this same gathering “Again I say to you, that if two of you shall consent upon earth, concerning any thing whatsoever they shall ask, it shall be done to them by my Father who is in heaven.” Interesting.
Your argument is like a man of whom it is written had handed the keys of his car to someone, and in a later passage we find that the man has allowed several others to drive his car as well, but there is no mention of him handing them the keys.