Reading this: (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)

You need a rdbms. Field names won't be repeated for each query and you will push later the time to 'start over'.