This is incorrect.
According to the Catechism of the Catholic Church, “***Divorce ***is a grave offense against the natural law” (CCC 2384). So, while there are surely victims of the mortal sin of divorce, not all those who divorce are innocent victims. In other words, there are those who deliberately commit the grave sin of divorce. So, divorce can be in and of itself a grave sin, even if not ever remarried.
Such grave sin, willfully and knowingly consented to can prevent a person from receiving Holy Communion, unless they receive sacramental absolution in the Sacrament of Confession.
It isn’t JUST remarriage that is a sin, but divorce itself is a grave sin. “Divorce is immoral also because it introduces disorder into the family and into society. This disorder brings grave harm to the deserted spouse, to children traumatized by the separation of their parents and often torn between them, and because of its contagious effect which makes it truly a plague on society.” (CCC 2385).
I think many Catholics have so focused on the problem of remarriage that they’ve lost the sense that divorce itself is a grave sin.