MaryEllen above has got it right, IMO. The fact that you’re questioning it and wondering is a good thing. Much better than to go on without questioning your thoughts, which if not tamed or discerned, can lead to sin.
I’ve almost always found that when I dislike a person due to “personality conflicts”, it’s usually something in my own personality that I am resentful towards. You may want to examine this and see if there’s something more there.
For example, at one place I worked, I absolutely detested one female coworker, because IMO she was so very vain and always looked perfect…her walking through the doors in the morning, ruined my day–but I just thought our personalities clashed because she was so FAKE. I later discerned 1) I was so jealous 2) I couldn’t afford to do the upkeep looking so great demanded 3) I didn’t have the time even if I had the money etc…
My point is, there’s a reason this person, regardless of whether he’s a priest or not, gets on your nerves. You might find more peace if you can get to the bottom of this.