A nativity set is art. It is a devotional item.
We must be careful not to speak of “should” and “shouldn’t” and “right” and “wrong” because there is no right and wrong.
The liturgy has rubrics. The GIRM and other Church documents specifies what may and may not be on the altar, said, sung, or otherwise used in the Mass and other liturgies.
The Church does not have rubrics on nativity scenes and other displays associated with the holiday season, especially those that aren’t even inside the church building.
Let’s not be scrupulous people. It is fine to have the nativity set up. It is fine to have baby Jesus in the nativity set. It is fine to have the nativity set without baby Jesus. It is fine to wait until Dec 24 or 25 to put the nativity set up. It is fine to have the wise men in the nativity set. It is fine to wait until Jan 6 to put the wise men in the nativity set. It is fine to take the nativity set down on Dec 26. It is fine to leave it up until Jan 6.
There is no right and wrong. Let’s not state our opinions on how we prefer to do things as if they were rubrics of the Church.