I think the concentration on the OP and her husband and the giver is missing something. And that something can be used in the response to the giver.
What is the most important thing here? That the daughter learns that whatever the intent, you don’t accept personal gifts from men outside the family. For the response, it really doesn’t matter at all what the giver’s intentions were. This is fortunate because nobody except him knows what they were.
The response I’d suggest is something along the lines of, “I’m sorry, and I appreciate your generosity, but we think it would set a bad precedent for our daughter to learn that it is OK to accept gifts like this from men. Of course, it is OK in this particular situation, but we find that, since she is too young to tell what is OK and what isn’t, we just have to have a “zero-tolerance” policy about gifts of that sort. It just wouldn’t be safe for her to think it is OK. I’m sure you understand, and I sincerely hope you won’t take offence.”
If he does take offence, too bad. It means he puts his ego over your daughter’s future safety. The most important thing is that your daughter (and any younger children, who will no doubt be paying attention) learns not to accept inappropriate gifts. It is a bit less important for them (right at this moment) to learn why. It would be a good idea to explain a bit later, late enough so that it doesn’t look like you’re saying anything bad about that particular giver, but definitely before she leaves home (for college, for example).
It is dangerous to set the precedent for accepting gifts like that, and it seriously limits your ability to refuse gifts on her behalf from less savory people in the future, because you won’t be able to refuse them without it sounding a lot like an accusation. (“Gosh, Mom! It was OK when so-and-so gave me those clothes! Why isn’t it OK for so-and-other-so to do it? It’s not fair, you’ve always been against so-and-other-so…” and so on.)
–Jen