Dear friend,
It is not a sin to wear the brown scapular without praying. But the whole point of wearing it is to express our devotion to Jesus and Mary. The scapular is a sign of such devotion. If the devotion is absent, then the wearing of the scapular makes no sense.
Fr. Vincent Serpa, O.P.
