You are free to attend any Catholic church recognized to be a Catholic church by the local diocese. Church shopping is discouraged because Catholics traditionally go to the Catholic church within their parish boundaries, but it is no longer required that they do so. But it is still a good idea to avoid the temptation of the cult of celebrity, in which people go to a church because they happen to like the pastor or avoid a church because they dislike him. If a particular priest is so problematic that he is a danger to your children’s faith, that might be reason to go elsewhere; but, if he is merely annoying on occasion, that might be reason to offer up your suffering and pray for him (and to teach your children to do the same).