I have the same problem, I see others do, too. I find going with someone helps, and going to daily Mass, so I can sit apart near the door, helps. Also, going early so I can pray before people show up.
The earliest Sunday Mass is the least-attended at my Parish. Mostly old folks. I used to, at my old Parish, like sitting in the front row all the way to the right because there was a door there and I faced the wall.
Sometimes we have to walk out. I think God understands completely. Don’t stop trying, though. You’ll find a way that works for you. Go when you can, go to Adoration if you can. A lot. And don’t beat yourself up about it or feel badly.
It gets better with age. God bless you abundantly.