I think that you should probably approach the issue from Mary’s point of reference instead of Jesus’s to being to get a good understanding of the situation.
In Jewish culture, Mary was a persona-non-grata. Period. Full stop. As horrifying as it may seem to us today, this was the case. The only thing that kept her from being stoned to death (as is also testified to in the protoevangelium by her trial of spirit, although her innocence was only revealed to the priests of the Temple of the time), was Joseph’s marrying her despite her appearance of infidelity. If you read Matthew 13:56 carefully, it only states that the sisters (that is, step-sisters through Joseph’s first marriage) reside in Nazareth. The people of Nazareth say they know Mary, but do not say that she lives there. It is most likely that she fled from Nazareth once Joseph passed away and, with him, his protection of Mary from the cruelty of the townsfolk. This seems to be the case, as some traditions have her either living with Mary, Martha and young Lazaras in Bethany or in Jerusalem in the same vein as the prophetess Anna, spending her days and nights in the Temple. She is seems to be outside of Nazareth when she and Jesus’s brothers are found outside the door and seek entrance (also note that the sisters aren’t there, reinforcing that this is not Nazareth).
Only unmarried women or widows without children were welcomed back into their maiden families. She was both a widow and had a child. As such, it was Joseph’s family who were required to take care of her. The sons of Joseph did not abandon her. It was just that, by Jewish law and custom, her status as a Widow with children would not allow them to take her into their homes. We know that James, the brother of Jesus was around, for he is identified as the apostle James, son of Alphaeus, the Lesser, also called the Just, second bishop of Jerusalem.
John, however, was a special case. He was young and most likely unmarried. He was not the eldest son (as James the Greater was still alive at this point) and as such did not have responsibilities to his own mother if something were to happen to his father. The acceptance of the care of Mary was wholly within his ability and it would have been allowed by Jewish custom as an act of compassion to take an unattached widow into his home as his mother.