A later Ecumenical Council went into detail that the Bishop of Rome was considered “first among equals” because Rome had been the Imperial City (not because of some unique succession from St. Peter in Rome as opposed to the successors of St. Peter in Antioch or the successors of the other Apostles in other places) and detailed that the Bishop of Constantinople was 2nd because it was the new Imperial City.
If it were because of some special thing to do with something unique about St. Peter, then the Bishop of Antioch would have been named 2nd (or 1st) since St. Peter founded the Church in Antioch & Rome and both cities had Bishops who were successors to St. Peter.
You are forgetting the reason that Catholics believe in the papacy (in reference to the office of the Roman Pontiff, not the Pope of Alexandria

) is because they see a need for a single, distinct sign of unity, for whatever reason, be they scriptural, historical, logical, fanciful or whatever. If you do not address this, you will have a hard time convincing Catholics of your position. The problem with most EO arguments is that they only attack the historical evidence for the papacy (alleging that it is not sufficient to justify modern papal prerogatives) without providing any clear positive alternative.
As for why Rome and not Antioch, we first have to acknowledge that according to the Catholic doctrine of the papacy, it could only be one or the other. There was only one Peter among the Twelve and similarly, according to the Catholic view, there can only one Peter among the bishops today. Since no one has ever asserted that Antioch was first in rank or that the bishop of Antioch possessed what Catholics say of the bishop of Rome, I don’t think this can be a forceful objection. Although the see of Antioch was always connected with the St. Peter, this was always in the context of the traditional ranking of Rome as first among all the sees, followed by Alexandria (also Petrine by way of St. Mark), and then by Antioch third. Before Constantinople asserted itself, this was the consensus of the Church and no one ever claimed that Antioch was first before Rome so how could it be a credible possibility today?
It is also true, as you said, that the Council of Constantinople and Chalcedon claimed authority for Constantinople on the basis of its political position. However the canons in question (chiefly Canon 28 of Chalcedon) were never accepted in the West, which you will see if you read the letters of Pope St. Leo. Therefore, these canons cannot have ecumenical status according to the position most commonly given by EO online because they do not have the consensus of the entire Church. Moreover, neither Constantinople or Chalcedon ever asserted that either Rome or Constantinople were “first among equals,” and the very point of Canon 28 of Chalcedon was to assert the prerogatives of Constantinople over other bishops. Canon 28 reads,
Following in all things the decisions of the holy Fathers, and acknowledging the canon, which has been just read, of the One Hundred and Fifty Bishops beloved-of-God (who assembled in the imperial city of Constantinople, which is New Rome, in the time of the Emperor Theodosius of happy memory), we also do enact and decree the same things concerning the privileges of the most holy Church of Constantinople, which is New Rome. For the Fathers rightly granted privileges to the throne of old Rome, because it was the royal city. And the One Hundred and Fifty most religious Bishops, actuated by the same consideration, gave equal privileges (ἴσα πρεσβεῖα) to the most holy throne of New Rome, justly judging that the city which is honoured with the Sovereignty and the Senate, and enjoys equal privileges with the old imperial Rome, should in ecclesiastical matters also be magnified as she is, and rank next after her; so that, in the Pontic, the Asian, and the Thracian dioceses, the metropolitans only and such bishops also of the Dioceses aforesaid as are among the barbarians, should be ordained by the aforesaid most holy throne of the most holy Church of Constantinople; every metropolitan of the aforesaid dioceses, together with the bishops of his province, ordaining his own provincial bishops, as has been declared by the divine canons; but that, as has been above said, the metropolitans of the aforesaid Dioceses should be ordained by the archbishop of Constantinople, after the proper elections have been held according to custom and have been reported to him.
Obviously, all the bishops mentioned cannot be equal since the canon states that the metropolitans are to be ordained by the bishop of Constantinople. This is a prerogative unique to Constantinople. It is a fact that the Church has always been structured in a hierarchical way such that not all bishops possess equal authority within the Church.