According to moral theology there are times when the seriousness of this sin is reduced. I would trust that your priest (who has no doubt studied moral theology) understands this much more deeply than any of us here and his advice should be trusted absolutely.
Moral theology manuals also point out to priests (specifically in this particular sin) that a person must be guided very gently because of the nature of the sin. It is a difficult habit to break and a priest doesn’t want the person to feel hopeless against it - because there is hope - it just takes time and a very concerted effort of the will.
For those who suffer from this difficult to conquer sin, the Church has, in the past, recommended things like:
Sleep on your right side with your arms folded across your chest
Get up immediately in the morning - don’t lie about in bed
Avoid thoughts of impurity or impure touches of other people (even the “safe” parts)
If you get thoughts of impurity, put them out of your mind immediately - think of something else
Additionally, you can consider joining the archconfraternity of the cord of St Joseph. You wear a cord with 7 knots around your waist at all times. It has indulgences and helps you to persevere in purity and chastity. A traditional Catholic priest should know how to make the cord and how to bless it and join you to the archconfraternity. If he doesn’t know, he will probably be able to find out for you. Here is more information on it:
fisheaters.com/stjosephcords.html