Careful with this one as it’s a bit incorrect: For me the location matters not, but the successor to St. Peter did take a small vacation in Avignon France.
True the vacation at first was not voluntary, Peter suffered a persecution from those who vied for his office and protesters to his divine authority from Jesus Christ. Fear of eminent death threats delayed his vacation at Avignon.
Let’s be honest to ourselves here; Forcing Peter out of Rome by persecution to take refuge in Avignon France, is not a vacation. And this persecution, is not the same as Peter leaving Rome or not remaining the bishop of Rome.
Although the cardinals loyal to Peter expressed their delights at the “vacation” in Avignon France, and wanted to remain there, probably the French has better cooks?
As stated by Steve VH, Peter remains the Bishop of Rome always while under persecution. no matter where his persecution takes him. This is a promise from Jesus to Peter; “They persecuted me, and they will persecute you…behold I will be with you always”
On a Technicality, Peter never leaves Rome, because both Peter and Paul’s bones are buried there to this day.