These folks just came on the scene two weeks ago spamming reddit's LocalLLama with the next clone of a llama.cpp wrapper like its a new thing. None of the featureset here is unique. An experienced developer could replicate this in hours in any language. More time was spent on the different permutations of their landing page than the actual app. Just go to /r/localllama and search for jan.ai and witness yourself.
Looks like they raised funding though. They're hiring.
Side-note: I really don't get the C++ in AI thing at all. It's becoming a meme. I wonder why they didn't go with Rust instead. It'll be easier to deploy to the edge as WASM, too.
Although Rust is an amazing language with rapid adoption, it still has a relatively smaller user base in low-level programming, where llama.cpp operates. As a result, the pool of talent that can contribute to such projects would be more limited. Almost all low-level programmers can write in C++ if needed, as for C programmers, it's essentially like writing C with classes. Rust programmers, especially those who care about low-level details including throughput and latency, almost always have a background in C or C++. If llama.cpp were written in Rust, there would likely be far fewer contributors. Considering that one needs to be at least interested in deep learning to contribute, the fact that it currently has 476 contributors is impressive. [1] I think this is one the most important reasons the project can move so fast and be such an essential project in the LLM scene.
Llama.cpp is a lot of C SIMD. I think using C++ elsewhere in the project just made sense in its infancy, when it was CPU only.
Rust is really interesting for stuff besides running the actual llm though. Python can be a huge performance/debugging pain when your frontend and such get huge.
My understanding is the proliferation of “XYZ-cpp” AI frameworks is due to the c++ support in Apple’s gpu library ‘Metal’, and the popularity of apple silicon for inference (and there are a few technical reasons for this): https://developer.apple.com/metal/cpp/
What does Rust offer here? More people know C++, security stuff from rust comes in the way of rapid exploratory development and arguably, C++ templates (don't think they are used here, moot point I know) are much more superior to rust.
