Matt 27:55-56
55 There were also many women there, looking on from afar, who had followed Jesus from Galilee, ministering to him; 56 among whom were Mary Mag’dalene, and Mary the mother of James and Joseph, and the mother of the sons of Zeb’edee (James and John "Boanerges

.
Joseph is also rendered “Joses”.
Mark 3:18
and James the son of Alphaeus,
So this James, the son of "the other Mary (also refered to as the sister of the Mother of Jesus), gave birth to James Joseph, Simon, Jude and their sisters.
The other James is the son of Zebedee, so maybe they think Mary the mother of Jesus was in cahouts with him?
Mark 1:19-20
19 And going on a little farther, he saw James the son of Zeb’edee and John his brother, who were in their boat mending the nets. 20 And immediately he called them; and they left their father Zeb’edee in the boat with the hired servants, and followed him.
Jerome writes:
“[Helvidius] produces Tertullian as a witness [to his view] and quotes Victorinus, bishop of Petavium. Of Tertullian, I say no more than that he did not belong to the Church. But as regards Victorinus, I assert what has already been proven from the gospel—that he [Victorinus] spoke of the brethren of the Lord not as being sons of Mary but brethren in the sense I have explained, that is to say, brethren in point of kinship, not by nature. [By discussing such things we] are . . . following the tiny streams of opinion. Might I not array against you the whole series of ancient writers? Ignatius, Polycarp, Irenaeus, Justin Martyr, and many other apostolic and eloquent men, who against [the heretics] Ebion, Theodotus of Byzantium, and Valentinus, held these same views and wrote volumes replete with wisdom. If you had ever read what they wrote, you would be a wiser man” (Against Helvidius: The Perpetual Virginity of Mary 19 [A.D. 383]).
I agree with Jerome that reading the Fathers makes one wise.
For those who are willing to read the background on the history of the perpetual virginity there is
a good article in the library here to get you started.
Or, you can follow the weak and superficial writers offered by rev kev. who do not include the history of this matter.