That’s my feeling on the matter. Poor health, poor food and genetics meant people were smaller than they are today. Heck, in Asian countries my 5’8" husband is a “giant”. In Australia, however, he feels surrounded by “giants”.
Given that Goliath was very physically capable and strong, it is unlikely he was much over 6 and a half feet or so—we know that much past that stature joints, muscles, and ligaments start to fail and most people become crippled or delicate at a young age when they are so structured.