Well - you have my sympathy, because I have been down that road, to my immense regret.
I had a spiritual director many years ago, a religious order priest, who one day announced that he’d rather be my “friend”.
Oh boy. I was flattered, shocked, attracted all in one hit, and I had never entertained any thoughts towards him before that point.
I am not married, and never have been, by the way.
So - these things can happen to us because we can be attracted by their spirituality, their pastoral concern for us as an individual, etc.
Just let the warning bells sound in your ears, and nip it in the bud!!
Go to confession (but not, I repeat NOT to this priest!) - and don’t be afraid to be fairly specific about what your problem is - a member of the clergy.
I’ve never had a spiritual director, since, BTW.

My prayers are with you!