Do you have specific sources that show evidence of this?
Here are some writings from the church fathers on Peter, the papacy, etc.
“The blessed apostles, then having founded and built the Church (in Rome), committed into the hands of Linus the office of the episcopate…To him succeeded Anacletus; and after him, in the third place from the apostles, Clement was allotted the bishopric…In the time of this Clement, no small dissension having occurred among the brethern at Corinth, the Church in Rome despatched a most powerful letter to the Corinthians…To this Clement there succeeded Evaristus. Alexander followed Evaristus; then…Sixtus (the list continues)… In this order, and by this succession, the ecclesiastical tradition from the apostles, and the preaching of the turth, have come down to us.” [Against Heresies III, 3, 3]
[T]he blessed Peter, the chosen, the preeminent, the first among the disciples, for whom alone with himself the Savior paid the tribute [Matt. 17:27], quickly grasped and understood their meaning. And what does he say? “Behold, we have left all and have followed you” Clement of Alexandria [Matt. 19:2 7, Mark 10:28] (Who is the Rich Man That is Saved? 21:3-5 [A.D. 200]).
In the power of the same Holy Spirit, Peter, both the chief of the apostles and the keeper of the keys of the kingdom of heaven, in the name of Christ healed Aeneas the paralytic at Lydda, which is now called Diospolis" Cyril of Jerusalem [Acts 9 ;3 2-3 4] (Catechetical Lectures 17;27 [A.D. 350]).
Be it known to you, my lord, that Simon [Peter], who, for the sake of the true faith, and the most sure foundation of his doctrine, was set apart to be the foundation of the Church, and for this end was by Jesus himself, with his truthful mouth, named Peter, the first-fruits of our Lord, the first of the apostles; to whom first the Father revealed the Son; whom the Christ, with good reason, blessed; the called, and elect (Letter of Clement to James 2 [A.D, 221])
"A question of no small importance arose at that time. For the parishes of all Asia, as from an older tradition, held that the fourteenth day of the moon, on which day the Jews were commanded to sacrifice the lamb, should be observed as the feast of the Saviour’s Passover. It was therefore necessary to end their fast on that day, whatever day of the week it should happen to be. But it was not the custom of the churches in the rest of the world to end it at this time, as they observed the practice which, from apostolic tradition, has prevailed to the present time, of terminating the fast on no other day than on that of the resurrection of our Saviour…Thereupon Victor, who presided over the church at Rome, immediately attempted to cut off from the common unity the parishes of all Asia, with the churches that agreed with them, as heterodox; and he wrote letters and declared all the brethren there wholly excommunicated.” Pope Victor & Easter (c. A.D. 195).
“And he says to him again after the resurrection, ‘Feed my sheep.’ It is on him that he builds the Church, and to him that he entrusts the sheep to feed. And although he assigns a like power to all the apostles, yet he founded a single Chair, thus establishing by his own authority the source and hallmark of the (Church’s) oneness. No doubt the others were all that Peter was, but a primacy is given to Peter, and it is (thus) made clear that there is but one flock which is to be fed by all the apostles in common accord. If a man does not hold fast to this oneness of Peter, does he imagine that he still holds the faith? If he deserts the Chair of Peter upon whom the Church was built, has he still confidence that he is in the Church?” Cyprian, The Unity of the Church, 4-5 (A.D. 251-256).
“After such things as these, moreover, they still dare–a false bishop having been appointed for them by, heretics–to set sail and to bear letters from schismatic and profane persons to the throne of Peter, and to the chief church whence priestly unity takes its source; and not to consider that these were the Romans whose faith was praised in the preaching of the apostle, to whom faithlessness could have no access.” Cyprian, To Cornelius, Epistle 54/59:14 (A.D. 252).
“Stephen, that he who so boasts of the place of his episcopate, and contends that he holds the succession from Peter, on whom the foundations of the Church were laid…Stephen, who announces that he holds by succession the throne of Peter.” Pope Stephen I [regn. A.D. 254-257], Firmilian to Cyprian, Epistle 74/75:17 (A.D. 256).
“Peter alone do I find— through (the mention of) his mother-in-law —to have been married. Monogamist I am led to presume him by consideration of the Church, which, built upon him, was destined to appoint every grade of her Order from monogamists.” Tertullian, On Monogamy ch. 8
“If you can travel into Asia, you have Ephesus. But if you are near Italy, you have Rome, where we also have an authority close at hand. What a happy church that is! On which the apostles poured out all their doctrine along with their blood: Where Peter had a like Passion with the Lord; where Paul has for his crown the same death as John.” Tertullain, The Prescription of Heretics, Ch. 36
[T]he Lord said to Peter, “On this rock I will build my Church, I have given you the keys of the kingdom of heaven [and] whatever you shall have bound or loosed on earth will be bound or loosed in heaven” [Matt. 16:18-19]. … Upon you, he says, I will build my Church; and I will give to you the keys, not to the Church; and whatever you shall have bound or you shall have loosed, not what they shall have bound or they shall have loosed” -Tertullian (Modesty 21:9-10 [A.D. 220]).
I sought out the full source for all of those that I could in order to try and understand more of what they were saying. I always worry that with short quotes it may be hard to understand the context. I am going to share the link, and sometimes a larger quote and some of my thoughts. I am trying to see clearly the different sides of this discussion.