Well, this is all speculative, but from what I have read, Jesus’ family was pretty much low-average by the standards of the time. They weren’t really poor.
The Holy Family had a roof over their head, probably had a reasonable amount to eat and wear.
They weren’t living in the streets or dependent on alms.
They had the means to rush off to Egypt on short notice (although one tradition says Joseph used the gold given as a gift by the Magi to finance that trip) and support themselves there.
Jesus had a father who had a skilled trade job and taught it to his son. Jesus also received a good education, and could read and write (as shown by him reading the Jewish scripture in the temple) and probably do math including geometry that he would need for his carpenter job.
The truly impoverished would have been people like widows with no son to provide for them, or disabled people who were begging in public in order to survive. Many of those people were also illiterate.
Elizabeth was probably a little more well-off because from what I have read, temple priests were the wealthiest and highest class in Jewish society of the time. However, there were several hundred priests and Zachariah was not of the top echelon, so he was probably what we would consider “middle class” or “high average” today. Which was still pretty good in a society where many people were begging in the street. Also, when Mary goes to visit Elizabeth, she is apparently welcome to stay for several months in Elizabeth’s household and they have no difficulty supporting an extra guest in the house for that time, so that tends to show they had some resources.
When we talk about Jesus and the Holy Family being “poor”, it means that they weren’t overly attached to or attracted to material things. They lived a simple life without luxuries and if someone had been in need, they would have helped the needy person rather than spent money on something lavish for themselves. Probably same for Elizabeth and her family.