Generally speaking it comes down to around what age children obtain the use of reason and can be consider morally responsible for their actions. Baptism is invalid if it is done against the will of someone who has the use of reason. Once we have obtained that ability then it must be respected and we should, in an age appropriate way, assent to our baptism.
At what point that happens is a matter of the prudential judgment of the Church. Its not dogmatic but the line has to be drawn somewhere. The Church, based on tradition and experience, has judged 7 years of age as the age when children can presumed to have begun the use of reason.
