This is a common misconception about how life expectancy works. If you have 10 children, 9 of them die at age 5, and the 10th dies at age 55, the life expectancy of your children is 10. Those 9 children who die early aren't going to go through puberty and become parents, but the life expectancy of those who do reach that stage is decently long. Low life expectancy doesn't mean you're considered some decrepit geriatric at age 12; it means that you get to grow up with lots of siblings who didn't make it.
When you look at pre-modern societies (essentially until around 1900 or so), there are four large causes of low life expectancy that don't really exist anymore. Infant mortality was extremely high (a quarter or so of live births didn't make it more than a few days). Then child mortality itself was high (you had perhaps a 50-50 chance of making it to puberty). For young women, there's an extra hit to life expectancy due to risky childbirth (you have about a 10% lifetime chance of dying in childbirth). For young men, they get extra risk due to endemic warfare--the baseline casualty rate of premodern warfare is basically WW2-levels of total war. Of all these factors, the infant mortality is the heaviest drag on life expectancy, as dying at age 0 really pulls down the average lifespan.
I think a more useful metric would be average life expectancy of the subset of people who lived over say 20 or 30. We could compare that number at 1600s with the present to see the net impact of lifestyle changes (fast food, minimum exercise) vs advances in medical care.
I think this is a case where an average value cloaks the distribution. Consider very high infant and childhood mortality rates, and how that might skew the average value.
Doesn't that sound extremely weird? Like, how does puberty, and 9 months to have a baby, fit in with a 12 year total life expectancy?
Maybe I need more coffee or something...