God made men and women for each other. Feeling of attraction towards other human beings is something completely natural and normal.
What I am worried about is how you say you’re indifferent. This might not be your case but a lot of people who feel this way usual suffer from some sort of childhood trauma that was caused by a female figure in your past (someone like your mom, aunt or sister) which has made you distance yourself and build a wall between you and other females. Sometimes this indifference could also arise due the absence of a father figure in people’s lives. Again, I’m not saying this is what is going on with you, but what I am trying to suggest is to look back on your life and childhood and see if their is anything that has caused you to distrust women and seal away your emotions. Are you perhaps afraid of someone getting to know your true self and then hurting you?
If all of the above is a big no, then the only other hint lies in what you just said - You feel this is a spiritual burden. Let’s elaborate on that. In what sense do you feel this is a spiritual burden? Are you saying that by getting to know her, you’d ,for whatever reason, end up distancing yourself from God?