Jesus ate with sinners. In a time when the Pharisees were trying to get all of the Jews living like righteous Jewish priests, in order to encourage God to send the Messiah, Jesus actually reduced dietary and behavioral strictures on His disciples, to less than what was expected of a Jewish non-priest. On the other hand, He taught much more severe strictures against hatred, envy, and other impurities of the heart and interior-based behavior toward others.
Of course, outward behavior in the Beatitudes sense and the Ten Commandments sense is still very important. But if Jesus freed us from Jewish kosher rules and hair rules and clothing rules, we can only voluntarily put ourselves under rules of the same kind about hair and clothes.
Wearing an impressive phylactery like a Pharisee, or wearing an impressively different outfit than what is normal, can be either spiritually helpful or a vanity, or even a burden bound on people that God did not demand. St. Francis de Sales said that a Christian layperson living in the world should dress in a normal way, and never be the worst-dressed or the most extravagantly fashionable person around. You dress with simple good taste, so that people understand that you are not a loony, and thus they take the example of your life and words more seriously.
(Obviously clergy and religious have a different job than laypeople, in his view, so people back then thought clergy usually should dress distinctively and religious should wear habits - like soldiers wear uniforms. Unless they were going undercover, of course, which he never had to do.)
You live in the freedom of the children of God. If people demand that you not eat with Gentiles, so to speak, you can tell 'em off, right to their face. Other than keeping our robes white, in a trying not to sin and going to Confession sense, and a general admonition not to dress in a way that is stupid immodest, we can do what we want or what is needed.
A lot of separatist religious movements essentially form lay non-Catholic religious orders, with any kids in the position of oblates like Samuel. It isn’t necessarily bad to want to “come out of Egypt” or “flee to the wilderness”; plenty of early Christians did it, and that is how monasticism started in the Egyptian desert. But it isn’t really acting on the lay vocation; it is being a monk.
Be nice about what you say, but be firm.
