You’re assuming things, however, when you start doing that. You’re assuming that things then were the same as today, so you use today as a template to explain how things were and developed into today.
Take, for example, a crater or large canyon. You note that rivers slowly engrave mini-canyons in the Earth. You therefore explain the large canyon by theorizing that a river slowly carved it out of the Earth. Judging by the breadth and depth of the canyon, you assume it must have taken millions of years for such a thing to take place, and therefore conclude that the Earth, also, must be at least that old. Of course, if there was some catastrophic event that caused a deluge, like a sudden melting of ice in the caps and Mountain peaks caused by an increase in temperatures due to enormous volcanic activity, for example, caused huge-wide scale flooding across the Earth, thereby expediting a natural process creating the appearance of age. What appeared to happen slowly in fact happened quickly. If the latter event occurred, then you’re assumption that the canyon was formed slowly over millions of years would be incorrect, as there is more than one way for things to form. You also have to assume you are aware of all the myriad possibilities that are causes for things to be. If aliens in some distant solar system tour the Earth once every thousand years in order to test their latest super-cannons for fun/target practice, then that could account for some of the craters, which to the aliens are more like pot-holes from their newest, latest bad-boy cannon
Of course, I am using science-fiction in the latter example only to demonstrate that it is quite foolish for us to claim authoritatively what happened and when in absence of a witness. We must assume that what we know presently is satisfactory for explaining the origin of things accurately and truthfully. As a result, any new theory or discovery will upset the
status quo as new data is considered for explaining past phenomena.
Hence an accurate historical account is necessary for science to determine origins; otherwise, we cannot know (based on observations alone) because we were not there to witness what actually happened. If we are correct, it is largely by accident, and if we are wrong, likewise, it is by accident.
The same goes with your age-of-man theory. If someone has a genetic disorder that causes him to age rapidly, then you’re ability to judge his age will be flawed, because you are not considering every possible factor that could create the appearance of his age. You therefore assume wrongly that a man is 40 when he may well only be 24. My best friend is 25; however, most people assume he is in his thirties because he had a disease as a young man that caused his bones and joints to grow too large, too fast, and he also started balding in his early 20s.
Had you bumped into Adam and Eve when they were first created, you would wrongly assume they were in there teens, twenties or thirties or older, when in fact they were only a few days old. They were created physically developed, endowed with some knowledge and with the appearance of age, all of which things would lead us to assume they were older than they in fact were.