It's extremely surprising that by 3pm in the middle of the day, there are as many people enjoying leisure time as there are people working (about 30%). Where are these people? I don't see anyone leaving the office at 3pm where I work. I'm pretty sure that the 'average day' doesn't mean weekend.
I thought it might be night shift workers, but then during the night, only about 1% of people are working.
I would have agreed with you before I got a job in the public sector. I'd say a quarter of the people in the buildings around here are gone by 3, with hardly anyone still here by 6. Some of those people start their day at 6 too, but there are also some "in by 10, out by 2" kinds of people. Nearby DC seems to be the same way from their rush hour traffic patterns.
The American Time Use Survey covers all the days of the week, so these numbers represent averages across all the days.[1]
If you work a five-day week with a two-day weekend, then 28% of the time you're also not working at 3PM, so don't get too jealous of all the lucky stiffs having 3PM leisure time. :)
I thought it might be night shift workers, but then during the night, only about 1% of people are working.