So, to you, having a desire to have child with a woman who is not your wife is not adultery ?
If I may rephrase, I would suggest that having a desire to take a woman to wife with a view towards an eventual family is not necessarily adultery.
As a (mostly) happily married man

I can recognise that other women are attractive, both in their physical appearance, and in their natures as people. Further I can recognise that the possibility of sharing a life and a family with some of these women is not entirely distasteful, Indeed for many years I harbored a desire to see polygamy introduced into the Catholic Church, reasoning (poorly) from St Paul that if love is truly not jealous, then my 1st wife shouldn’t have a problem with it. After all the only scripture I could find against polygamy was ‘man shall not serve two masters’ - and I’m not sure that was meant to be applied to two wives, and ‘the two shall become one’ - and then that composite one and another one shall become what? 1.5?
What convinced me that polygamy was inappropriate for any true follower of Christ, was Humane Vitae, and it’s description of the nature of marital love. This is not something distasteful, or something dirty, but something beautiful, a gift from above, a gift that once received should be held so dearly ‘forsaking all others’ not deeming all others to be somehow unworthy, but recognising that the road before you is so special that one chooses not to follow any other not because there is anything intrinsically wrong with each of those roads not taken, but because changing mid journey would be disrespectful to the gift already received.
Similarly for the priesthood, celibacy is not an assertion that women or marriage are somehow dirty or distasteful, rather that they are a good that one freely gives up in service of the Christ.
I realise I mashed up a few metaphors there but I hope you grasp my meaning.
Regards Doc