Who cares if people think you’re a religious nut? Their opinion of you means NOTHING compared to God’s opnion of you. Do not be ashamed of Him and do not prefer the praise of mere worldly mortals.
As Pope Leo XIII said:
Truly, he who disdains the worthless judgments of the mob, who prefers to undergo the scourging of insults rather than abandon duty in any matter, proves himself to be of a far greater and exalted spirit
True humilty involves not being ashamed of your faith. Remember, whoever denies Jesus before men will be denied by Him before the Father.
Likewise, this is a good opportunity to share your faith. If people ask, explain to them why we fast and do penance. Who knows, you may plant some seeds.
If you feel worried, pray this prayer before entering:
O Jesus, meek and humble of heart,
Hear me.
From the desire of being esteemed,
Deliver me, O Jesus.
From the desire of being loved,
Deliver me, O Jesus.
From the desire of being extolled,
Deliver me, O Jesus.
From the desire of being honored,
Deliver me, O Jesus.
From the desire of being praised,
Deliver me, O Jesus.
From the desire of being preferred to others,
Deliver me, O Jesus.
From the desire of being consulted,
Deliver me, O Jesus.
From the desire of being approved,
Deliver me, O Jesus.
From the fear of being humiliated,
Deliver me, O Jesus.
From the fear of being despised,
Deliver me, O Jesus.
From the fear of suffering rebukes,
Deliver me, O Jesus.
From the fear of being calumniated,
Deliver me, O Jesus.
From the fear of being forgotten,
Deliver me, O Jesus.
From the fear of being ridiculed,
Deliver me, O Jesus.
From the fear of being wronged,
Deliver me, O Jesus.
From the fear of being suspected,
Deliver me, O Jesus.
That others may be loved more than I,
Jesus, grant me the grace to desire it.
That others may be esteemed more than I,
Jesus, grant me the grace to desire it.
That, in the opinion of the world,
others may increase and I may decrease,
Jesus, grant me the grace to desire it.
That others may be chosen and I set aside,
Jesus, grant me the grace to desire it.
That others may be praised and I go unnoticed,
Jesus, grant me the grace to desire it.
That others may be preferred to me in everything,
Jesus, grant me the grace to desire it.
That others may become holier than I,
provided that I may become as holy as I should,
Jesus, grant me the grace to desire it.