I'm Brazilian and when I moved back home for a bit the country had recently inaugurated a national exam for all high-schoolers. About maybe a year after, they'd publish an anonymized data set for researchers with a nice question-by-question breakdown of how every student performed. I worked at a school at the time, and one thing we wanted to know was how WE had performed in the questions so we could gauge if there were some things our students were consistently unprepared for.
They didn't provide that data, but it turns out with a little bit of grokking and staring at that 10gb text file, you could reverse engineer it so you could extract all the kids of a given school, and aggregate all of their answers. I produced a nice little report for our admins, with the questions of the text next to how we had performed in the aggregate and state averages, as well as averages of our "competitor schools."
The best part, though, is how I remember it being a bit "bullshit" that we, a private school, could afford to do this, but since the data was actually valuable to inform practice, surely the department of education should do that for every school! Whelp, over the weekend, I computed that info for every school in the dataset, and just stored a CSV for every school in an S3 instance (this was my ridiculous caching strategy lol). Spun up a frontend where you could select your school, and nicely visually go through every question, as well as print a pdf summary of the whole thing. On Monday morning tweeted at an ed journalist, and in a few days had a spread of me in the country's top newspaper, and people emailing about jobs "at my company."
This was the most rewarding project I've ever done, and I'm sad to say nothing has come close since. It cost me $0, produced a public good that I could see was being accessed from every state in the whole country, was technically interesting, and I saw it through from start to finish over the weekend!
Great work! Inspiring for me, as I am trying to build a better UI to consult the document “Base Nacional Comum Curricular” and help all schools to use it for their curriculum creation.
Sadly, no. I moved back to the US and then worked on other things, but still maintained it for a few years until the ENEM microdata started to get released super behind schedule, then I gave up on the whole thing. Was called dadosdoenem.org, and you probably may encounter a reference to it here or there somewhere online. Good times... insane to think it was over a decade ago.
They didn't provide that data, but it turns out with a little bit of grokking and staring at that 10gb text file, you could reverse engineer it so you could extract all the kids of a given school, and aggregate all of their answers. I produced a nice little report for our admins, with the questions of the text next to how we had performed in the aggregate and state averages, as well as averages of our "competitor schools."
The best part, though, is how I remember it being a bit "bullshit" that we, a private school, could afford to do this, but since the data was actually valuable to inform practice, surely the department of education should do that for every school! Whelp, over the weekend, I computed that info for every school in the dataset, and just stored a CSV for every school in an S3 instance (this was my ridiculous caching strategy lol). Spun up a frontend where you could select your school, and nicely visually go through every question, as well as print a pdf summary of the whole thing. On Monday morning tweeted at an ed journalist, and in a few days had a spread of me in the country's top newspaper, and people emailing about jobs "at my company."
This was the most rewarding project I've ever done, and I'm sad to say nothing has come close since. It cost me $0, produced a public good that I could see was being accessed from every state in the whole country, was technically interesting, and I saw it through from start to finish over the weekend!