Here are his words:
“In one place I said… that the Church had been built on Peter as the
Rock… but in fact it was not said to Peter, “Thou art the Rock,” but rather “Thou art Peter.” The Rock was Jesus Christ, Peter having confessed Him as all the Church confesses Him, He was then called Peter, “the Rock”… (ed, for his faith) …Between these two sentiments let the reader choose the most probable.” (St. Augustine, Retractions - 13th Sermon; Contra Julianum 1:13)
St. Augustine also adds: "Peter had not a primacy over the apostles, but among the apostles, and Christ said to them “I will build upon Myself, I will not be built upon thee.” (ibid.)
In still another letter Augustine quotes Cyprian, with whom he is in full
agreement:
“For neither did Peter whom the Lord chose… when Paul afterwards disputed with him… claim or assume anything and arrogantly to himself, so as to say that he held a primacy and should rather be obeyed by newcomers…”
Finally, Augustine concludes, near the end of his earthly life, with these words on the “Rock of the Church”:
“Christ said to Peter… I will build thee upon Myself, I will not be built
upon thee. Those who wished to be built among men said, ‘I am of Paul, I am of Apollos, I am of Cephas’ - however, those who did not wish to be built upon Peter but upon the Rock say, I am of Jesus Christ.” (Retractions, 13th Sermon)