First of all open mind one thing that needs to be made clear here is that in Baha’i and Islamic theology what we are experiencing here in our physical bodies is a “lesser” reality to the spiritual reality which is in comparison, “true reality”
We are spiritual beings having a physical experience.
So when you say, “the fact that the soul is unaffected is a separate thing” you have made the first error. The soul is unaffected because the soul is the TRUE REALITY. So if a baby is to lose out on its physical experience, the soul will still exist, actually in a more perfected condition than my soul, for example.
Let me explain.
In the Baha’i Writings (without using flowery language

) you will see the soul of man metaphorically described as a “mirror”. God is REPRESENTED by the Messenger (Jesus, Muhammad, the Bab, Baha’u’llah etc) and we can metaphorically describe Him as the Sun.
When the spark of faith is lit, our mirrors (souls) are TURNED towards the Sun (e.g Baha’u’llah) and reflect its light. We are now showered with all the godly grace and divine susceptibilities.
But, we do have another nature to our humanity, the animalistic, ego-driven aspect, and this is metaphorically described as “dust”.
When we allow the dust, using our free will, to settle upon our mirror, the light of the Sun no longer shines so resplendently, and this reduces the intimacy of our union with our Lord.
When we die, we leave with our mirror in a beautiful, radiant condition, adorned with all the “light” attributes of our Lord, or less so depending on the dust and dross that we have allowed to settle upon our mirrors (souls)
In Baha’i theology, a baby is born with a perfectly clean mirror (the soul is of God and is radiant). Were the baby to die, it will not have had much time to allow the dust of ego, greed and desire to settle upon it. Perfect justice is therefore automated. The baby has no physical experience, but it lives the true reality in the spiritual realm as close as one can get to communing with our Creator.
Now God, does not necessarily want this to happen. He wants to test His servants, and that is the purpose of this physical “experience”
"For with fire we test the gold, and with gold we test Our servants" - Baha’u’llah
However, why do babies die at a premature age?
We are told that whatever happens on the physical realm is as a result of some form of human intervention, sometimes the intervention that causes early death is discovered and known, sometimes the intervention is an unknown one.
But, what Baha’is do know is that any intervention that is outside the scope of the Ordinances/Teachings and Exhortations of the Manifestation of God or Messenger of that age is the root cause of ALL human suffering.
For example, in this day, Baha’u’llah has forbidden the drinking of alcohol. I remember when I was a teenager, everyone looked at me strangely when I said I do not drink. Today, the validity of this teaching of Baha’u’llah is becoming more and more obvious, not only to our own physical health, but more importantly to societal health.
Now, if a teenager dies because he drank himself to death on the weekend, does that make it God’s fault, or the fact that this goes “against” the exhortations of Baha’u’llah?
If a baby dies in the mothers womb because the mother was getting drunk every weekend, is that God’s fault or the fault of turning away from Baha’u’llah’s teachings?
So you see, we are at fault for all our own physical problems.
But Baha’u’llah says that there are unavoidable victims to this “turning away” from God, and the baby in the womb is an example of this, but God in His perfect wisdom has endowed the perfect system whereby justice is always maintained for the wellbeing, ESPECIALLY, the spiritual wellbeing of the victims, which is of paramount importance…
I hope and pray that this assists your search for answers
.