For a long time, I pretty much refused to go to confession with any priest I personally knew. I’d wonder “are they thinking about what I did?” when I’d see them at Mass, parish functions, etc. (even though I know this is completely not true!) But one time, during a communal penance service, I decided to be “brave” and go to my own pastor. Now I’d prefer not to go to anyone else! It’s great to have a confessor who knows about my struggles, current things happening in my life, etc. If there’s a particular area where I keep stumbling, he knows the background and can advise me accordingly.
I concur with others here–it doesn’t matter where you go or who your confessor is. The important thing is that you are receiving the sacrament and its graces.