It is not a sin to fail to accomplish a penance… even though that wasn’t a penance anyway (rather, the priest was telling you what you SHOULD be doing anyway, which is avoiding that, and all, sin in the future).
In fact, let’s say you were legitimately forgetful and failed to ever do your penance… or you became incapacitated before accomplishing it… what would happen? Well, penance is for remission of temporal effects of sin, it has a purifying aspect to it, so you might spend a little more time in purgatory because you missed out on penance, but that’s about it.
If you were going to confession with the deliberate intent NOT to do any penance, nor with any desire for purification and abstintion from sin, I would say that this would greatly call into question your contrition… but that matter is for each individual to weigh in their hearts.