Easy enough to explain, if you like, using the example of Fanny Alger. Emma caught young Fanny and Smith in the barn. Fanny, who was living with the Smith’s, later began to show her pregnancy. Emma threw her out of the house. She left to live with relative, when she returned…no child.
Other women were married to other men, so technically, the women were polyanderous. Any child that was Smith’s was raised as the other man’s. Smith also “married” other men’s wives who were already pregnant. And then there is evidence for the practice of abortion in Nauvoo.
At a time when a pregnancy out of wedlock was not socially acceptable, women and their families already had the social construct in place to support women and girls who became pregnant outside of marriage. I don’t see “no children” as an argument.
Also, the idea that Smith married women for some reason that was never explained, is weak. It isn’t difficult to understand why a man would chase after women outside of his marriage.
As for “good fruits”, I don’t know that Mormonism has anything unique going for it. I know among Mormons there is a general feeling and teaching of “we’re special”, but you have to understand, no one else thinks that about you (as a group).
As for infallible humans, the difference with Smith is, he defined his sin under new doctrines and claimed these doctrines came from God. No prophet has ever done this, and I hope you understand why. It is a glaring, overwhelming, piece of evidence of what sort of “prophet” Smith was. False.