There is simply no reliable answer to how old St. Joseph was at the birth of Jesus. Scripture does not give us any real information to judge from and the non-canonical Gospels are not necessarily reliable.
The general consensus is that he was an older man due to the non-canonical writings using that tradition. The disappearance of him from the life of Jesus in the canonical Gospels leads many to believe that he had died before Jesus began His public ministry, thus adding to the idea that he was an older man.
However, in the 1st century AD people could drop dead at any age for any reason so the fact that St. Joseph may have died before Jesus’ public ministry doesn’t necessarily mean he was an older man.
I would hesitate to say that St. Joseph was a younger man only because the tradition of him as an older man seems so overwhelming. But, on the other hand, it is not a sacred tradition that cannot be questioned.
I would encourage you to ask the leader what the basis for his opinion is and then judge it based upon that…
