I have always been thought the star appeared when Christ was born. Thus it would have taken them some time after His birth to arrive as they would have been travelling with a caravan and not as three lonely travelers. After all three men alone would have been exceptionally prone to being robbed.
That being said there is further evidence that time had passed because Matthew says that Herod asked them when the star appeared and then, when he realized the Magi had left by another way, sent his troops to slay all the males under two years.
So clearly it seems that Jesus, who would have been presented and circumcised on the eighth day, would have had plenty of time to have been circumcised, gone back, raised to an age old enough to travel safely and then flee with his father and mother.
There is further evidence from extra Biblical sources with Josephus declaring Herod had died around 1BC, and not 4BC as later translations have put it, and the census was taken around 5BC, then that would mean Jesus was actually born around 4BC, which is where many Biblical scholars date it as well as astronomical researchers who put the “Star” which was possibly a convergence with Jupiter, and a star in Aries (the Ram being the astrological sign for Israel) occurred late in the year 5BC into early 4BC which would also line up. Of course some do not think this was the “star” and I don’t know but thought it was interesting.
Anyway if He was born in four, the Magi come in 3/2 Herod dies when He is 3/4 years old then He returns passing by Bethlehem at 4/5 years old.
Lines up pretty well I think.
God Bless