First of all, a single person, never married, can certainly commit adultery.
What is important here is Matthew 5.1: “But I say to you that whosoever shall put away his wife…maketh [or causes] her to commit adultery.” This concerns culpability (complete consent), one of the conditions for a sin to be a mortal sin. And there is this from Matthew 19,9: “And I say to you, that whosoever shall put away his wife and marry another, commits adultery against her; and he that shall marry her that is put away, committeth adultery.” What does that tell you?
Where does it say that she that is put away commits adultery? Isn’t it said she is made or caused to commit adultery? Where then is culpability? If someone holds a gun to a person’s head and makes or causes them to jaywalk, they may be technically guilty of jaywalking but are they culpable?
If Jesus says “whoever divorces his wife” how then is divorce unreal or impossible? What is it the guilty spouse–in your first sentence above–is guilty of if it isn’t divorcing his spouse? How would the fact that “the guilty divorcer cannot marry again” support you argument re the innocent spouse? And there is in your last sentence: “Not on whether someone was responsible for the divorce…” What divorce? The impossible one?
And what might this mean: "In fidelity to the words of Christ - “Whoever divorceshis wife and marries another, commits adultery against her; and if she divorces her husband and marries another, she commits adultery” (CCC 1650). The law of contradiction is certainly in play here. Divorce is a reality, and it is a reality the Church recognizes. So is a second marriage. These are facts, not impossibilities. It is why the divorced and remarried, absent an annulment, are not permitted to receive communion (absent the exception of living as brother and sister). That the first marriage would remain valid in the eyes of the Church is also a fact. Is this what you mean by “contradiction”? It is the teaching of the Church.