I agree with you that priests need time to themselves and a chance to relax. It’s a hard life. But it is the life that God has called them to. They are called to the priesthood and the trappings of the priesthood is a prominent position within the community. In an ideal world it would be
the prominent position in the community.
Yes priests need “downtime”. This can be facilitated by retreats and vacations.
But I am not going to ignore the laity’s role. We
shouldn’t be pestering a priest whilst he’s collecting milk. I love our priests but quite frankly I’m not going to ask him for his views on Molinism as we stand in the dairy isle.

I’ll pause to say hello out of politeness and respect and inquire after him. But I wouldn’t want to harass him and sadly many people do. The answer is not that priests should ditch their clerics; laymen just need to learn some common courtesy. Indeed, priests should also be able to relax around laymen and have “downtime”. I’ve knocked back far too much whisky than I care to count with priests who had no compulsion about wearing their clerics whilst doing so.

And I’ve never yet experienced a Polish priest that would let the collar halt the copious amounts of alcohol going down his throat as he enlightens you on the finer aspects of his old pastor back in the forests that spent more timing out hunting than saying Mass!
Priests are men and need time to just be normal guys watching the football, sharing drinks and conversation or watching Star Trek marathons (that particular priestly-friend is a bit of a nerd.

). But they can do all of that whilst wearing their ecclesiastical garb.
I would like to note again that I am not saying that priests need to wear a cassock at all times. Certainly it’s desirable and I would like to see every priest wearing one even if the diocese needs to fund it. But they should still wear clothes identifying them as clergy whether that is a cassock or a shirt and collar with slacks. Say a priest needs three cassocks. That’s US$628. He doesn’t need to buy them all at once. I’m sure a priest could comfortably save up for that over a period of time.
Ad Jesum per Mariam!
Prie-dieu.