The simple answer is that with or without what you do or do not recognize as “lust,” it is the turning inward of a gift (sexuality) that is given by God and meant to be used outwardly, within marriage, rather than directed toward the self.
While it is also a physical urge, it is, in the human being, much more than that. We are meant to engage our wills regarding our sexual urges, as opposed to simply surrendering to these biological urges and letting them control us, as if we are lower animals.
Sexual activity is integral to our dignity as full human beings, and thus it is related to our ability to love another person (not just ourselves) and to create life with a
partner.
We all have many appetites; some of these are more prominent in certain phases of our life and at certain ages. We have difficulty controlling some of our behavioral impulses as young children, sexual impulses in adolescence (and some would say throughout our lifetimes

), verbal impulses – which some people substitute for physical violence – in adulthood, etc. In order to grow in spirituality we have to take control of our impulses, urges, and appetites – whatever those are and whenever they occur.
I’ve met some holy people in my life, yet never have met anyone of any age who had all of his or her impulses and appetites in complete submission to his or her will. Everybody is ultimately ‘on the journey’ and we all need each other’s prayers and support in that journey.