Can you sell or re-sell colo space to a handful of customers who might put TT or other “weird” hardware there? Doesn’t scale, but it hedges your own business. Requires the right customer though, like somebody who might buy Nervana / Xeon Phi but then buy NVidia from you when it blows up.
Kind of. It is easier for us to be the capex/opex (meaning buy and run equipment) and then rent it out. I have the backing to make large investments without too much drama. We can do that with any hardware as long as we have longer (1-2 year) contracts in place. We have the space/power available in a top tier data center (Switch).
That’s indeed critical, but most Director-level managers and below have very little control of how well the business model serves the OKRs. Yes the OKRs need to be achieved and help make the business work, but e.g. if the business model’s margins are just too tepid or if the VC’s expected revenue growth (exponential?) will never actually realize, then there is really zero material value to the shipped product. Hence the focus on a happy team that’s shipping, because at least that provides some technological value. And build a network you can bring to your next gig—- because that’s what gets you the next job.
There are rare cases where a team might discover a new business model or impress a whale customer, and then the business model fundamentally changes.
Yes there is risk the “bean counters” or CFO / COO office will want to cut the cord (especially now tech hiring is in a recession). But tech moves fast; those bean counters will likely end up owning shares of a zombie in the next 5-7 years. And their game is to cash out, not build a future.
And if the business model actually works, then keep at those OKRs and everybody should win. Good business models are where stupid can succeed; the team has the right levers.
The main idea behind DSPy is that you can’t modify the weights, but you can perhaps modify the prompts. DSPy’s original primary customer was multi-llm-agent systems where you have a chain / graph of LLM calls (perhaps mostly or all to OpenAI GPT) and you have some metric (perhaps vague) that you want to increase. While the idea may seem a bit weird, there have been various success stories, such as a UoT team winning medical-notes-oriented competition using DSPy https://arxiv.org/html/2404.14544v1
There are a handful of good restaurants (e.g. Sea Pal Cove) and a nice off-leash beach beach there despite it being a very sleepy corner of the small city.
They claim unlimited access, but in practice couldn't a user wrap an API around the app and use it for a service? Or perhaps the client effectively throttles use pretty aggressively?
Interesting to compare this $200 pricing with the recent launch of Amazon Nova, which has not-equivalent-but-impressive performance for 1/10th the cost per million tokens. (Or perhaps OpenAI "shipmas" will include a competing product in the next few days, hence Amazon released early?)
Discord servers are great, especially if they are tied to an open source project for grounding. Good ones might be hard to find; you might have to try one that isn’t directly related to your interests for a while until somebody there links you to a community or other Discord who has folks you would want to follow.
Branching out like this is critically important even if your company has senior folks who can give you helpful feedback, since your own company will have its own biases.
The result is minor AND Google spent a (relative) lot of money to achieve it (especially in the eyes of the new CFO). Jeff Dean is desperately trying to save the prestige of the research (in a very insular, Google-y way) because he wants to save the 2017-era economically-not-viable blue sky culture where Tensorflow & the TPU flourished and the transformer was born. But the reality is that Google’s core businesses are under attack (anti-trust, Jedi Blue etc), the TPU now has zero chance versus NVidia, and Google is literally no longer growing ads. His financing is about to pop in the next 1-2 years.
What makes you say TPU has zero chance against growing NVIDIA?
If anything, now is the best time for TPU to grow and I'd say investing in TPU gave Google an edge. There is no other large scale LLM that was trained on anything but NVIDIA GPUs. Gemini is the only exception. Every big company is scrambling to make their own hardware in the AI era while Google already has it.
Everyone I know who worked with TPUs loves how well they scale. Sure Jax has a learning curve but it's not a problem, especially given the performance advantages it gives.
Besides the many CAPEX-vs-OPEX tradeoffs that are completely unavailable due to not being able to buy physical TPU pods, there are inherent Google-y risks e.g. risk of the TPU product and/or support getting killed or fragmented / deprecated (very very common with Google), your data & traffic must also be locked in to Google’s pricing, and you must indefinitely put up with / negotiate with Google Cloud people (in my experience at multiple companies: worst customer support ever).
Google does indeed lock in their own ROI with deciding to not compete with AMD / Graphcore etc, but that also rooflines their total market. If they were to come up with a compelling Android-based Jetson-like edge product, and if demand for said product eclipses total GPU demand (robotics explosion?) then they might have a ramp to compete with NVidia. But the USB TPUs and phone accelerators today are just toys. And toys go to the Google graveyard, because Googlers don’t build gardens they treat everything like toys and throw them away when they get bored.
Not a full solution, but seeing the OP seeks to be a key-value store (versus full RDBMS? despite the comparisons with Spanner and Postgres?), important to weigh how Rockset (also mainly KV store) dealt with S3-backed caching at scale:
By chance has your employer posted on HN who’s hiring?