Even if there is in some particular case where there is discerned an exception by the priest regarding an invalidly remarried person - even they would need to be in a state of grace.
Therein lies the problem.
How does a remarried person (without an annulment) who deliberately chooses to continue in the second conjugal union, achieve the state of grace? This deliberation is not bypassed or excepted by “discernment” or “discussion on the internal forum,” but it is rather intensified. With all this discernment and discussion, certainly one has the freedom and knowledge to realize that these acts are grave matter. Then in choosing to persist, the choice is solidified.
It is impossible then, unless during all this mysterious “discernment” - which nobody can explain clearly - some distressing factor persists, i.e. “He will beat me to death,” “She will take all of my money and my children,” etc. (Having a sex drive is NOT a distressing factor.)
On the other hand, one might realize that he or she is likely to sin again but sincerely intends not to. This person is getting there and would even be in a state of grace, provided there is a good confession.
Additionally, there is the case of foreseeing certain situations in which there would be little or no culpability for the act itself which is then intended habitually - and this is almost always on the part of the woman being coerced through fear of some kind. Could fear of the loss of temporal goods be sufficient to excuse one through distress? Only up to a point - and that point is the one where reason regains control over fear. (This is different for women, because they can simply allow the act to occur, while men basically can’t but must commission it themselves.)
Finally, there is the question of normal sexual temptations between persons in a second union who are striving to live as brother and sister. Probably there could be some cases where an individual act occurred without grave sin - but if steps are not taken to minimize temptation, one’s true heart is slowly revealed while the will becomes weaker and the intellect darker on the path back to a clearly and intentionally adulterous relationship. This could also reveal a secret pride, and is itself a temptation to breed habitual carelessness for offending God - and that is directly against charity.
Mad times we are living in. Post-Fall, that is.