That idea stems from the (to be fair, historically inaccurate) popular Christian belief - since the 2nd century Protoevangelium of James - that Zechariah was actually a high priest. From 4th century that we see different Christian writers expressing that Zechariah was high priest and received his annunciation on the Holy of Holies, either on Yom Kippur (the Day of Atonement) or on Sukkot (Feast of Booths), which they often confuse with each other.
You might say that Luke somewhat encourages this idea in a way, because he has Zechariah burn incense, writes as if Zechariah was the only person within the sanctuary, and mentions people praying outside the sanctuary. Automatically his readers’ minds would default to the biblical descriptions of the high priest offering incense by himself during the Day of Atonement.
But while Luke could very well have described Zechariah’s actions as if they were the actions of the high priest during Yom Kippur, but it’s not automatic indication that Zechariah was historically a high priest. In other words, Luke’s narrative is possibly more theological than literal here. In any case, we have no records from Josephus of a high priest named Zechariah during the relevant time period (7-4 BC, the last years of Herod’s reign): the high priest during that time would have been either Simon son of Boethus (reigned 23-5 BC), Matthias son of Theophilus (reigned 5-4 BC) or Joazar son of Boethus (Matthias’s wife’s brother, reigned 4 BC). Not to mention, as you mention a plain reading of Luke does not give any hint that he was either the high priest or that it was the Day of Atonement; incense was offered in the temple everyday at morning and at twilight. And while only Zechariah and the angel is described in the scene, it does not necessarily have to follow that Zechariah was alone in the sanctuary.
The Mishnah specifies that at least five priests were required for the daily service: those who tended the menorah - the lampstand - and prepared the altar, the one who will offer incense, and his assistant. After doing their job, said five priests would go outside the sanctuary, standing on the steps of the porch and, along with other priests, confer the priestly blessing (Numbers 6:24-26) on the people. If we suppose this tradition is accurate, then Zechariah could have been one of these five priests; the reason the people would have been surprised at his delay (1:21) may have been because only four of the five priests came out.
