I’m going to gently disagree with you here. I know this is a sensitive topic for you and very personal.
I’ll start by saying that I think it’s awful that people have said things like that to you. I think people go to family size as a conversation topic to try and find common ground, but then falter when it doesn’t go as planned. Most of the time, I think people mean well, but that doesn’t make it less awkward or hurtful in the moment. And of course, there are the few who mean for their words to hurt, but can pretend like they don’t when called on it, and that’s worse.
That said, I
do think there are certain difficulties that are more likely to arise in families with only one child, and they do include some of the things you mentioned. But most parents work very intently on making up for those issues. My husband was an only, but his parents fostered really close connections with first and second cousins and had regular opportunities for him to socialize with other children at a young age. I think that was really great for him. It’s not a “given,” but I do think given the dynamic that there are certain things that are more likely to arise.
(And the same goes for larger families, too - there are certain things parents need to be careful of and be proactive about when it comes to forming their children, and they’re going to have to be ready to respond to the situation as is. Commentary from strangers is probably inevitable, even though it’s inappropriate, and parents in this day and age have to develop a bit of a filter to sort out good advice from busybodying.)
The Church does ask us to be generous with regards to family size. Siblings ARE a blessing. But not everybody is called to give that, for a variety of reasons - sometimes it’s not possible, sometimes it just isn’t very prudent. And in highly dysfunctional families, yes, sometimes certain relationships can bring a lot of pain. I am not especially close to my siblings as an adult, for a variety of reasons. But I can still recognize their God-given human dignity and yes, there is an intimacy we have simply because we were all raised together. I don’t think it does well to dismiss that as unimportant, either.
Personally, I disagree with your assessment that there are statistically significant numbers of people who decide to have another child primarily due to social pressure. I could see that being an element, but I think for most people, in functional families, they get through the infancy period, realize they’re capable, and like sex enough to say, “What the heck? We’ll give it another go.”

They may desire to give their child a playmate or sibling, but because of the inherent goodness in that, not that other people are crowing at them about it.
As I said, I know this is sensitive for you, and I’m not trying to be hurtful. I certainly don’t wish to tell you to have another child or to get involved in your parenting decisions. But I also want to caution you to not throw away the wisdom of the Church, even if in your particular situation it’s not the best course of action.