Insensibility is a vice opposed to chastity, just like lust.
The fact is that by signing up for marriage, you are signing away the rights over your body. Paul says it clearly… See 1 Corinthians 7.
There must be a proportionately grave reason to deny your spouse’s right to your body… If it’s just a fleeting moment of desire, that’s one thing, but it is another if it is a more serious request. For what it’s worth, St. Thomas did not even think leprosy would count as sufficient for denying a truly serious request, though it would suffice for not living under the same roof.
You’re bound to get a lot of objections to this, about how that’s “objectifying” and “crazy” or what have you, but I think it is missing the point (marriage is designed by concession - to help with the vice of lust! - or else we should all just be celibate or at least perfectly and perpetually continent in marriage), and by entering marriage one does indeed become an “object” insofar as one becomes one flesh with the spouse… (Besisdes, this neo-Kantian and personalist language is, to be honest, not always the most useful…) While one’s own body is not an “object,” it is also subject to one’s own will. Of course, each spouse remains a person, but with regards to sexuality, it is given over to the use of the other spouse, by his or her will. Ideally, it is not an issue - but it is indeed an issue for some people who have bad communication skills or are just not having a happy marriage. But the rights and duties are the rights and duties.