There aren’t really enough facts in the original post to be able to answer. What is a weekly and monthly schedule like? What is your work schedule like? Do you have an 8 - 5 job and are home every evening and every weekend? Do you have any hobbies? If so, how much time do you spend on them in a week/month? Does your work take you away from home overnight? If so, how often in a month are you gone, and for how long?
If this is the only activity that you engage in in a month without the family, and all other evenings and weekneds you are home with your family, then I would suggest the following: ask your wife in a nice way if you can have your pants back, as she seems to be wearing them.
If, however, this is just one more thing that takes you away from family, and you are gone several (that is, more than 2, but not many) nights each week, then perhaps you need to take a look at your schedule and cut some things or re-arrange them.