Baby showers, whether co-ed or all-women, are usually for close friends and family.
Unless the priest has some connection to your family besides just being parish priest, such as he’s the baby’s uncle, the dad’s best friend from college, the mom’s boss at work, etc. then I wouldn’t invite him to the baby shower.
As someone else said, he’ll be at the baby’s baptism and you can certainly invite him to stop by for cake or whatever after that ceremony.