Objectively speaking, it doesn’t matter a single bit.
Subjectively speaking, you need to examine your conscience. If you were invited to be greeted personally in a public audience by the pope, would you feel bad doing so in what you wear to mass? If yes, you aren’t dressed properly. The pope is just the Vicar of Christ. Christ HIMSELF is coming to you at mass.
I rarely wear a tie to mass, but do so at work. I consider that different because I am COMPELLED to do so at work. I consider the dang thing a leash.
I’ve also worn jeans (neat and clean ones) as penance for catching myself feeling smug about being better dressed than others at mass. Dangerous sentiment.
I’ve said this on other threads - the only reason I would dress up for a Pope, King or President is that they are frail human beings who, sadly but true, have our common human tendency to judge at least partially by appearances.
That being the case, unless I were in an especially contrarian mood I would tend to make allowances for this and dress up at least a little, depending on the occasion of course. Going to an outdoor barbecue on the beach, even with the Pope, wouldn’t call for a three-piece suit I should think.
With God I highly doubt there is any need to appeal to the external in the same way. Didn’t Our Lord say John the Baptist, who wore rough-and-ready camels-hair clothing, was the greatest of men?
And didn’t He also say something against those who wore broad phylacteries and long fringes on the prayer-shawls (in other words who dressed ostentatiously for prayer or worship) since the right spirit was just as likely to be absent from their prayer as anyone else’s?
In summary - clean clothes? Absolutely. Not torn or immodestly tight or brief or sheer? Without question.
Not rumpled? If possible (some of mine I would swear simply cannot exist in a crease-free state for more than 30 seconds). Goes without saying no offensive pictures or slogans or other design features.
Preferably no pictures or slogans of any kind, since it can be a bit distracting to look at what is essentially a walking advertisement for a clothing brand/drink/town/rock band/what-have-you.