The word “brother” is used here just as it is in the Gospel, where Our Lord says “who is my mother, who are my brothers?”
The Greek word for brother is adelphos, and in the New Testament, this takes on a much broader meaning than “brother”.
Note that in Aramaic and Hebrew, the native tongues of the writers, there is no word for “cousin” or other more distant blood relations other than by circumlocution (the son of my aunt); these are clumsy, so the Jews often used “brother”. Note, though, that there is indeed a Greek word for cousin, anepsios but the Septuagint, the Greek Old Testament from which 3/4 of the NT’s Old Testament quotes come from, use adelphos to describe cousins. It was almost an unwritten rule of style for a Jew writing in Greek to use “adelphos” for other relationships.
Our Lord has four “brothers” James, Joseph, Simon, and Jude. These were possibly cousins, or even step-brothers from a previous marriage of St. Joseph, who some people speculate to have been an old widower when Our Lady was to marry him.
Either way, they could not have been His brothers. First, Jesus entrusts His mother to John the Beloved. If he had brothers, His mother would have defaulted on them (and James the Less certainly would have been honoured to have looked after Our Lady). However, he choses a caretaker for her as if he were an only child, something that simply doesn’t make sense, and for all legal purposes could not have happened if he weren’t an only child.
Also, in the Gospels, Jesus’s “brothers” try to advise him. We are told that Jesus is the “firstborn” and if he had any siblings, they would have been younger. Now, amongst the Jews, younger siblings did not tell the older one what to do, especially the oldest.
This is only a snippet of what can be said in defense of the Catholic view. Rest assured that Our Lord was an only child, and that Our Lady was, is, and ever shall be, a perpetual virgin.
James certainly would have been a relative of Jesus, and as Bishop of Jerusalem, he had strong ties to the Holy Family and the Jewish roots of Jesus, he was not His biological brother anymore than my cat is.
Happy reading,
Mat.