So I am getting most of this from Tim Staples “Behold Your Mother.” Your best bet would be to look at the book yourself, but here is the cliff notes version for your argument.
Ok, so this is mostly focusing on the verse Matthew 13:55. Let us look at these brothers, James, Joseph, Simon, and Judas. One of the brothers mentioned is James, whom we see St. Paul referring to as a brother of the Lord, and one of the apostles (Galations 1:17-19). There are two James who are apostles, James the son of Zebedee (not the one mentioned in the passage in Matthew, because he was dead by the time Paul wrote the letter to the Galations, according to Acts 12:1-2). Therefore it must be James the son of Alphaeus (Luke 6:15-16) or Clopas (Matthew 27:56). Clopas is a Greek spelling of the Aramaic name Alphaeus, so this is the same James. Joseph, James’ brother is also mentioned as the son of Mary, wife of Clopas, whom John calls Jesus’ mother’s sister (John 19:25). Matthew 27:56 refers to these men, Jesus’ “brothers” again, as sons of Mary, the wife of Clopas. Now if we look at Jude 1:1, we see that Jude or Judas is the uterine brother of James, who we have just proven not to be a literal uterine brother of the Lord. So, three of the four brothers of the Lord can be easily shown not to be uterine brothers, but more likely cousins or close relatives. So let us then consider Simon. There is no biblical proof of this claim, but a second century historian by the name of Hegesippus reports that Clopas is the brother of Joseph (Jesus’ adoptive father) and that he had a son named Simon. This can be found as referenced by Eusebius of Cesarea in Eclesiastical History, 4:22:4 and 3:11:1.
Hope this helps.