In the government’s eyes, marriage is a form of contract or mutual agreement. It will never be extended to things or people that are incapable of entering into agreements or contracts. You will never have bestial marriage in the US because animals cannot consent to contracts. You will never have inanimate object marriage in the US because inanimate objects cannot consent to contracts.
I don’t think there is a good constitutional argument against polygamy, but the pragmatic one is simply this:
Allowing polygamy requires non-trivial changes to the rights and responsibilities of marriage as conferred by the government.
To legalize same-sex marriage, all you have to do is eliminate reference to gender, from the rights and responsibilities. Since most of the rights and responsibilities were gender neutral to begin with (e.g. tax free inheritance of spouse’s property) making this change was trivial.
But there is no trivial way to update the responsibilities for polygamy. You can’t just say “everyone gets to make everyone else’s medical decisions” and have that be the end of it. You need to actually pick a method to decide who gets power of attorney.
Because the decisions are no longer trivial, polygamists will have a divided front, and be significantly less likely to actually change their legal status.