Your history is a bit undeveloped.
First, you are talking only about the West. Secondly, even in the West, skirts on men, like kilts, are not unknown. Robes as well. In some periods within the last 500 years, men have worn “bloomers” that showed off their legs - in order to be sexy, essentially - lacy frothy clothes, make-up, wigs, and high heeled shoes. They were not trying to look like women though, that was concidered manly.
The Church says women should not try to be men, or be mistaken as men, and vice versa. We are to understand that our sex is part of who we are, and that it has certian implications.
Various cultures have signals of clothing, behavior, or whatever that follow gender - that is ok as long as they are not sinful in themselves, or used to oppress or hurt other people. So there is really nothing wrong with a culture where men wear beards and women cover their hair. A person living in those places should respect those norms.
In the West, the norms have changed, recently. So things are a bit mixed. There are other norms related to sexuality that have changed too, some good and some not. Denial of gender differences - not good. Women are people under the law and not property - good. A mixed bag, and it’s hard to see how it will come out, since we are rather in the middle of it.
So should an individual wear pants? Well, I would say, in the West, that one can wear pants without scandalizing social norms, or denying gender differences. So pants are ok. There are those who equate pants with wholly negative changes in our ideas about gender, but I think the changes are not wholly negative, so that doesn’t wash with me. Others suggest that pants are more revealing - also not convincing, there are many that are, just as many skirts and dresses are. (This too is partly cultural leanings - what is a sexual signal is learned, as ankles were in the Victorian period. And there are no clothes that will totally swallow sexual differences. Even in a burka, we’d notice a man’s broad shoulders and a woman’s smooth walk.)
And there is no good argument to say that pants, per se, are male garments. They are just a particular formation of cloth.