Homophobe is like “Racist” - in a vacuum, the word itself doesn’t need to be loaded one way or another. Racists once ran political campaigns touting themselves as “racists” before society caught up to it.
If you have negative attitudes towards homosexuals and homosexuality, be they feelings of contempt, or hatred, or prejudice, or antipathy, or inherent superiority, then you are a homophobe.If you have a problem with being called a homophobe, despite meeting the plain text definition of being one, that is, by definition, your problem. It’s not a pejorative term because I’m intending it to be one. It’s pejorative because you are supplying the negativity that follows the word around. And you can put it an end to it, simply by owning it. In some circles, “hacker” is considered a negative thing. In others, it’s a positive thing, and computer nerds are quick to own it. You can do the same thing with homophobe. You know, if not feeling an aversion to homosexuality is too tall an order.