That’s not exactly what grave matter means. Grave matter refers to the objective evil of the action itself. Murder and abortion are grave matter. Bank robbery and sacrilege are grave matter. Child abuse is grave matter. Adultery and fornication are grave matter. Purposely destroying someone’s reputation is grave matter. Get the idea - they’re all seriously bad things. Grave matter is the first of three requirements for a sin to be mortal.
The second is technically called “advertance,” which means you understand that the action is seriously wrong. I think this is what you were incorrectly calling grave matter. And you must understand at the time you commit the act that it’s seriously wrong. Finding out later does not count as advertance for the times you did the thing without knowing it was wrong. Once you know, of course, you would have advertance, but not the first time, before you knew.
The third is consent. You must decide freely to do the action in spite of its being seriously wrong. You can’t be doing it because you are coerced or fearing for your life, or otherwise not free to decide.
So, in your example, an argument or mean comment don’t always qualify as grave matter, so if you have advertance and consent, you’ve probably got yourself a venial sin.
Hope this helps.
Betsy