Not necessarily, like other things, these beliefs develop over time and would not have been present or widely held in the early Church.
While the fact that Jesus entrusted Mary to John on the cross could imply that he had no other siblings, he could also have just preferred that John take care of her and not his siblings.
Then there’s the fact that the brothers accompanied Jesus & Mary to Capernaum after the wedding at Cana (John 2:12 ) which implies them all being children of Mary (or possibly step-children from an earlier marriage of Joseph’s).
But it definitely is not a clear cut issue

We don’t have enough evidence to know either way.