I really feel for you. My DH was in a similar situation with his first wife - she had an affair and left him.
Divorce, in and of itself, is not wrong for you. The Church’s position is that a civil divorce does not end a marriage covenant, but it is not wrong for one to seek a civil divorce as a measure of protection in cases such as abuse, adultery, or abandonment. It would ONLY be problematic if you wanted to remarry without getting a declaration of nullity.
I strongly, strongly echo the others in encouraging you to consider approaching your priest to discuss a declaration of nullity. This is a statement from the Church that there was no valid marriage covenant in the first place. It does NOT deny that you had a relationship and a conjugal life; rather, it simply states that the nature of that relationship was not a valid marriage covenant. Many people find the nullity process very healing. A declaration of nullity also does NOT “end” the marriage; rather, it states that there were one or more reasons for the marriage not being valid in the first place (a couple that may apply here include not being open to children, and, possibly, not intending to be faithful).
Even if you don’t wish to remarry, the nullity process may still be very cathartic for you. There is a group on CAF for individuals who are seeking declarations of nullity, and there is an ongoing thread on the Family Life forum for those in this position as well. There is a member named Dan who is a canon lawyer and is a member of both groups; he is very well-informed on the nullity process and often provides advice to those confused about it. DH went through the nullity process and we have been sacramentally married for 2.5 years.