Don’t you care how someone dresses when attending an important meeting with you? If a date showed up wearing sloppy sweats and ratty flip-flops, what do you think it would say about that person’s attitude toward your date? If you were interviewing someone who showed up in short-shorts, a spaghetti-strap halter, and barefoot, would you give that person a job?
If someone dresses as nicely as he can for Mass but cannot afford better, that is one thing. But if someone deliberately dresses in such a manner that he indicates he doesn’t think Mass is important and doesn’t care what God and neighbor think about it, then yes, I think such an attitude might offend our Lord.