The word miter (or mitre) in English is derived directly from the Greek and Latin word* mitra*, meaning crown. There are many uses in the Greek New Testament of mitra. We see in 2 Timothy 4:8 and 1 Peter 5:4 that on judgment day crowns will be awarded and in James 1:12 and Revelvation 2:10 that these crowns are a reward for perseverance.
The miter’s origins in the Church aren’t precisely clear, but we do know, from the New Testament itself, that men were wearing some sort of headdress, at times inappropriately, in the early Church. In 1 Corinthians 11:4 Paul says at prayer men are to remove their hats, a rubric that continues to this day. (Men remove their hats as they enter a church, and a bishop removes his miter when he prays at a liturgy).
If we look at pre-Christian times, we see crowns, headdresses, and turbans of various sorts described and prescribed. Speaking of the high priest, Exodus 29:6 and Leviticus 8:9 say, “Put the miter on his head” In Exodus 28:3-4 the Old Testament is emphatic on what a high priest is to wear: "Y]ou shall give instructions to make such vestments for Aaron as will set him apart for his sacred service as my priest." These vestments include a linen miter (Ex. 39:28).
The miter of a Catholic bishop may not be derived specifically from this, but it does recollect it. We see a continuity, not necessarily in style, but in meaning. The miter is a ceremonial headdress for prelates which only the pope, cardinals, and bishops (all of them “higher priests,” though not the High Priest) can wear.
There are a few mentions of miters among early Church writers. Tertullian, writing about 220 in his De corona, refers to crowned apostles, evangelists, and bishops. The apostles James and John were said to have worn a headdress like that of the high priests of the Old Testament.
Eusebius, in his History of the Church (III, 31), speaks about the apostle John and says “Again there is John, who leant back on the Lord’s breast, and who became a priest wearing the miter.” James appears to have worn a priest’s miter, according to Epiphanius and Hegesippus.