Secure hardware sounds like the wrong idea, I think. I think the correct idea will be something more similar to block chains. A system where the security of the system lies in the ability for anyone to make a copy of the voting data at any point in time. So there will be multiple copies of the voting data, owned both by the authorities and by ordinary people.

If the authorities try to tamper with the central copy of the voting data, it will be checked by the multiple copies owned by the general public.

I think that's the general idea one should pursue. Not "secure hardware".