The SSPX does not deny that the Pope is the Vicar of Christ.
You must be confusing them with Sedevacantists, who deny at least the current pope.
You are right that they do not publicly declare the current, or the last 4 Popes to legitimately be Pope.
However, there are certain basic issues at play. Without getting side-tracked over issues of whether or not Vatican 2 was infallible, at the very least it was an act of the Magisterium of the Church; statements by the Pope (in union with the bishops, but I won’t even side-track that point) concerning Faith and Morals.
Either the Holy Spirit protects the Church and guides the Pope when he speaks officially concerning Faith and Morals, or the Holy Spirit doesn’t.
If we say “doesn’t” we can all go home and spend our time on something else.
If we say “does”, then the Holy Spirit had to be protecting the Church and guiding the Pope when the documents were signed and promulgated.
To deny that the documents of Vatican 2 are consonant with prior doctrine puts one into the position of either denying doctrine (that the Holy Spirit protects the Church) or saying that the Pope who signed (and those since who uphold the documents) committed heresy - which then leads logically to the sede position. There simply is no other way around it. One can use all sorts of high falutin language, but after one gets past all the obfuscations of it, that is where one is left.
Of course it is possible to play all sorts of word games about the issue and persuade a lot of people.
The very bottom line with the SSPX, which started with Archbishop Lefebvre, is the issue of obedience. Either the SSPX will obey the requirements of the Pope, or they will not. So far, they will not.