Even if it was not cost-effective, or you're just running worse models, it's learning an important skill.
Take for instance, self-hosting your website may have all these considerations, but you're getting information from the LLMs. It would be helpful to know that the LLM is in your control.
My self hosted website that has a bunch of articles about my self hosted setup and documentation, that i have running from a server in my living room got me my 6 figure devOps job.
> Self hosting website as local server running in your room? what’s the point?
It depends on how important the web site is and what the purpose is. Personal blog (Ghost, WordPress)? File sharing (Nextcloud)? Document collaboration (Etherpad)? Media (Jellyfin)? Why _not_ run it from your room with a reverse proxy? You're paying for internet anyway.
> Same with LLMs, you can use providers who don’t log requests and SOC2 compliant.
Sure. Until they change their minds and decide they want to, or until they go belly-up because they didn't monetize you.
> Small models that run locally is a waste of time as they don’t have adequate value compared to larger models.
A small model won't get you GPT-4o value, but for coding, simple image generation, story prompts, "how long should I boil an egg" questions, etc. it'll do just fine, and it's yours. As a bonus, while a lot of energy went into creating the models you'd use, you're saving a lot of energy using them compared to asking the giant models simple questions.
Considering how most major providers got their data to begin with these are EULAs that are probably impossible to enforce, and none of the major players have given much reason to believe they would stick to it anyway. And why bother when local models don't have nearly the same complications?
Right. And the former CEO of Google basically explained their operating procedure. You go and steal everything, you break the rules, "then you have your lawyers clean all that up" when you get caught.
> ...if you're a Silicon Valley entrepreneur, which hopefully all of you will be, is if it took off then you'd hire a whole bunch of lawyers to go clean the mess up, right. But if nobody uses your product it doesn't matter that you stole all the content.
Yeah. Because I have tons and tons of fear, uncertainty and doubt about how these giga-entities control so many aspects of my life. Maybe I shouldn't be spreading it but on the other hand they are so massively huge that any FUD I spread will have absolutely no effect on how they operate. If you want to smell real FUD go sniff Microsoft[1].
I run LLMs locally on a 2080TI I bought used years ago for another deep learning project. It's not the fastest thing in the world, but adequate for running 8B models. 70B models technically work but are too slow to realistically use them.
