The Brown Scapular is a private devotion which you are free to follow or not. So, if you don’t do part of the devotion there is no sin involved. You don’t have to go to confession for that. If you have been generally slothful in your spiritual life, that you should confess that, but following or not following a private devotions isn’t a sin. So, just do what the priest said and start again, if you wish. Private devotions are meant to be aids to our faith, not burdens to be borne. If a devotion helps you keep your baptismal vows, fine, do it. If it doesn’t help you, you don’t have to do that devotion.
The best way to help your spiritual life is to at least pray part of the Liturgy of the Hours (besides the Mass, its the official daily prayer of the Church). Morning and Evening Prayer is usually best for lay people to do:
divineoffice.org/ (scroll down past their pitch for donatons). Also, reading the daily Mass readings is a good way to feed your soul and keep up with the Church’s seasons, solemnities, feasts, etc. Just a suggestion for you to consider.