The sin that is chosen and not repented of is worse than the sin followed by repentance and amendment, but there isn’t anything in Holy Scripture that sets adultery above fornication or fornication above adultery.
The person who knows before marriage that his or her spouse would not be making vows as a virgin might have to work hard to get past that spouse’s past–it can be harder than it looks!–but such persons do not have carte blanche to punish a spouse for a sin that he or she implied was forgiven. Canon law reflects this in how it handles the innocent spouse’s rights when it comes to infidelity after marriage:
Can. 1152 §1. Although it is earnestly recommended that a spouse, moved by Christian charity and concerned for the good of the family, not refuse forgiveness to an adulterous partner and not disrupt conjugal life, nevertheless, if the spouse did not condone the fault of the other expressly or tacitly, the spouse has the right to sever conjugal living unless the spouse consented to the adultery, gave cause for it, or also committed adultery.
**Can. 1155 **The innocent spouse laudably can readmit the other spouse to conjugal life; in this case the innocent spouse renounces the right to separate.
It is wrong to alternatively give and then withdraw forgiveness for a past offense (provided, for example, it is not found out that repentance had always been false, of course). So while I’d say the adultery was not automatically worse than the fornication and while I admit that grief over a spouse’s unchastity can be very complicated and difficult to cope with, there is no excuse for holding past bad acts over someone’s head. That is not the same as re-adjusting one’s trust to a more appropriate level. That is not to say that extending real forgiveness for infidelity is something that can be done instantly, with no time allowed for the work and suffering involved. I mean tormenting the repented spouse over something he or she has amended to the best of their ability. Get past it or let them past you, so they can at least have peace; those are your choices.
Of course it is never OK to use someone else’s previous sin as an excuse to sin against them in a similar manner. That one doesn’t fly; it is totally opposed to the mind of Christ.