If anyone wants to try it with an esp32:

https://github.com/DaveBben/esp32-llm

needs a board with PSRAM but they are surprisingly plenty these days - almost standard. I tried it a few months back.