First of all, don’t let other people’s struggles, as seen from your point of view and not theirs, influence your decisions. Do your parents feel like bringing up 3 children was as difficult as you think it was? Why not ask how they did it and what it meant to them? You might get a very different perspective.
I married later in life at 35. I too got questions tossed at me at holiday gatherings in my 20’s. I’d just say that if I married it would be at the right time to the right person and that was more in God’s hands than mine. That was usually answer enough.

As someone else said, you have to set up boundaries, and not let others cross them no matter how well-intentioned they are. On way to do that is to physically remove yourself from them, in a nice way, until they stop pestering you.
As it happened, my dh and couldn’t have children, and I wasn’t keen on having any, as it was. Still, I’ve learned as a married person that whatever God sends my way, as long as I am doing his will everything works out just fine. I am content–not because we never have problems or struggles, everyone does no matter what life they choose, but because our trust is in God not in our circumstances.
So, whatever you decide to do, just be sure you are doing what God wants you to do. No matter who doesn’t understand or criticizes, you can have peace of mind and heart knowing that God will take care of everything that needs taking care of.