What I have heard (and it makes sense) is that in Jewish society, a woman was either under the ‘headship’ of her father, her oldest male relative, her husband, or her son. A woman who had none of the male protection above would find herself out on the street and forced into slavery or prostitution in order to survive.
So if Mary’s father had died before she reached puberty and she had no living male relatives (Elizabeth was her kinswoman, not Zachariah), then it would have been arranged, possibly by her father before he died, or by the temple authorities) for Mary to be betrothed. Her spouse would then be responsible for her.
Now if Mary had made a vow to live as a consecrated virgin, her father, relatives, or husband could do one of two things: they could have her cancel the vow (since they had authority over her) or they could let the vow ‘stand’ (thereby giving the vow ‘their authority’ to proceed).
So if Joseph were requested to be Mary’s responsible party, he could let the vow stand or have her consummate the marriage, even if he had previously said he would let the vow stand. If he DID consummate the marriage he was perfectly within his legal right, but he would be ‘looked down on’ by those who knew that he had originally agreed to something different.
This would explain a lot about the views of the people in Nazareth about Jesus --pride in Him but also a rather contemptuous view of Him and His family --if they thought that Mary had broken her vow, that she and Joseph had let their ‘lusts’ overcome a sacred vow. Jesus would be seen as of course a legitimate child, but a child of ‘profane lust’. They wouldn’t be surprised that He ‘broke out’ years later as a ‘meglomaniac’. And they would be sneering, “That son of Mary”. . .and his ‘ragtag’ family. They also would not have been surprised that Jesus was the ONLY son of Mary, as they would think that Mary and Joseph, after their lapse, would try to ‘make up for it’, although ‘of course’ they’d still be scandalous. The Blessed Virgin very likely had a lot of ‘barbs’ thrown her way by women who looked down on her. . . and Joseph probably had to stand a lot of male chaffing and being looked down on as well. . .and both knowing their innocence but submitting to the slander just as their Son would years later.