Since you make a distinction between your penance and the priest’s words about the rosary, it appears to me that the daily rosary was just a suggestion by the priest. If you are unsure, you could ask the priest to clarify whether the daily rosary was your penance. If it is not given to you as a penance, then you are free to take or leave the priest’s suggestions for spiritual improvement. It would be prudent though to explain to the priest that you don’t feel able to pray the rosary “daily” and ask if there are smaller steps you could take to start.
