Right – there’s not anything intrinsically sinful about not fasting (although you would be denying yourself the graces from it and also the valuable lesson in self-control).
Rather, the grave sin is in the blatant disobedience to our Mother the Church, who was given authority by Jesus Himself.
If a mother told a child to carry his hamper down to the laundry room, and he refused, would that be sinful? If so, is it because there’s something sinful about a hamper sitting around? No – it’s because of the kid’s disregard for his mother.