What we have here is a distinction between what is valid and what is proper (i.e. licit). Let me use an example. Technically I can say validly say Mass anywhere if I have the correct matter and form. However, the dignity of the sacrament requires that unless there is extenuating circumstances that I do so in a church. I shouldn’t just say Mass in my car because I can.
In the same way, the sacrament of baptism brings us into God’s holy Church, it makes brothers and sisters of Jesus Christ, sons and daughters of God the Father. The sacrament is deserving of dignity, honor, and an appropriate setting. A sacrament is never just a “me and God” moment because the entire Church is present and acting whenever a sacrament is celebrated. The external fullness of that is when the sacrament is celebrated according to the external norms in a church (Can. 857 §1).
As you point out, in danger of death or reasonable concern for survival, anyone can baptize but that doesn’t mean anyone should baptize at any moment.
