I’m not a fan of Extropic, but I’m seeing a lot of misconceptions here.
They’re not building “a better rng”- they’re building a way to bake probabilistic models into hardware and then run inference on them using random fluctuations. Theoretically this means much faster inference for things like PGMs.
Skimmed the litepaper. Has the flavor of: you can do "simulated" annealing by literally annealing. I like the idea of using raw physics as a "hardware" accelerator, i.e. analog computing. fwiw, quantum computing can be seen as a form of analog computing.
I do think that a "better rng" can be interesting and useful in and of itself.
Thanks for the Normal Computing post, it felt more substantial.
We experimented with doing ML training with it, but it's not clear that it trains any better than a non-broken PRNG. It might be fun to feed the output into stable diffusion and see how cool the pictures are, though.
Cloud RNG number streaming is interesting but costly, no? I did have the idea to serve truly random numbers via a quantum computer (trivial by just preparing the simplest state and measuring). Anything else can't be said to be truly random.
You don't need a quantum computer to sample noise from quantum processes.
It's a prohibitively expensive way to go, and depending on how you built the quantum computer, it may be more susceptible to interference and non-quantum noise than using good circuits and custom systems.
Oh sure, it was more of a joke app idea, or in another sense, for those who want philosophically "perfect" randomness. After all, assuming a single bit sample, anything but a hadamard state has nonoptimal entropy. Of course, it makes almost no sense to sample a bit at a time since we want pragmatism. But having a virtual coin flip essentially "create" a new, nonexistent bit of information in the universe -- that's funny (well, assuming certain interpretations of QM).
I'm using the term "analog computing" to mean using non-digital (or even digital but in a nonstandard way). A quantum processor is not digital as each qubit has an uncountable number of states. A quantum computer would likely have a classical (digital) part to measure the quantum processor's registers, which are a bunch of "analog" states. And even if one wants to use terminology in a way that qc isn't analog, it would still objectively share many qualities with "normal" analog computing (basically all of its differentiators against classical compute).
Then again, I should probably also ask what you mean by "analog computing", and why you think quantum error correction would not allow qc to be classified as analog.
For context, I did research on crafting a high-fidelity (error-correcting) quantum gate (successfully).
It did make me curious however, if we dropped the requirement that operations return correct values in favor of probably correct values - would we see any material computing gains in hardware? Large neural models are intrinsically error correcting and stochastic.
I’m unfortunately not familiar enough with hardware to weigh in.
The trouble is if you use actual randomness then you lose repeatability which is an incredibly useful property of computers. Have fun debugging that!
What you want is low precision with stochastic rounding. Graphcore's IPUs have that and it's a really great feature. It lets you use really low precision number formats but effectively "dithers" the error. Same thing as dithering images or noise shaping audio.
Yeah debugging would be a pain, but in the context of inference/training unnecessary. There is some set of ops which requires high precision, if I L2 normalize a tensor - I really need it to be normalized. But matmul/addition? Maybe there is wiggle room.
Big challenge would be whether any gains could compete with the economy of scale from NVidia.
I wouldn't want to write this off because you get the feeling these guys are on to something that could be hugely important (ignoring quantum this thermodynamic that) - but surely it feels like they need to get to the point a lot faster e.g.
"We're taking a new approach to building chips for AI because transistors can't get any smaller."
I really don't know what they gain by convoluting the point and it's pretty hard to follow what the CEO is talking about half the time.
Quantum computing people have been selling this exact spiel (including the convoluted talking points) for decades and it keeps working at getting funded. It has not produced any results for the rest of us, though.
One difference is that baking mathematical models into electronic analogs is older than integrated circuits. The reason we deviated from that model is because the re-programmability and cost of general purpose, digital computers was way more economical than bespoke hardware for expensive and temperamental single purpose analog computers. The unit economics basically killed analog computing. What Extropic (and others) have identified is that in the case of machine learning, the pendulum might have to swing back because we do have a large scale need for bespoke hardware. We'll see if they're right.
Quantum computing has been exploring an entirely new model of computation for which it's hard to even articulate the problems it can solve. Whereas using analog computers in place of digital is already well defined.
Arguably yes, in a commercial sense. To give the fusion folks some credit, they haven't been promising that commercial products are "just around the corner" for the last 30 years the way QC people have, and the quacks (cold fusion) were excised from the field for making those false promises. I do think that if your field as a whole continually makes huge promises and never delivers, it should probably tarnish that field's reputation.
However, if you're thinking about research grants, no. That's the point of research grants.
The tech could be really cool if e.g. classifiers could be represented within the probability space modeled on their hardware. However their shaman-speak isn't confidence inducing.
Your summary seems to miss a later quote from the article:
> Extropic is also building semiconductor devices that operate at room temperature to extend our reach to a larger market. These devices trade the Josephson junction for the transistor. Doing so sacrifices some energy efficiency compared to superconducting devices. In exchange, it allows one to build them using standard manufacturing processes and supply chains, unlocking massive scale.
So, their mass-market device is going to be based on transistors.
The actual article read like a weird mesh of techno-babble and startup-evangelism to me. I can't judge if what they are suggesting is vaporware or hyperbole. This is one of those cases where they are either way ahead of my own thinking or they are trying to bamboozle me with jargon.
I personally find it hard to categorize a lot of AI hype into "worth actually looking into" vs. "total waste of time". The best I can do in this case is suspend my judgement and if they come up again with something more substantive than a rambling post then I can always readjust.
Am I the only one who thought the article was clear, lucid, and reasonably concise?
The company's success or failure will depend on execution, but the value proposition is quite sound. Maybe I've just spent too much time in the intersection between information theory, thermodynamics, and signal processing...
"Don't splurge on high SNR ('digital') hardware just to re-introduce noise later." == "Don't dig a hole and fill it in again. You waste energy twice!"
> Doing so sacrifices some energy efficiency compared to superconducting devices.
In most applications superconductivity does not actually yield better energy efficiency at system level, since it turns out cooling stuff to negative several hundred degrees is quite energy demanding.
I don't disagree. I just come away from the article feeling more confused as opposed to enlightened and excited about what they're building.
It even makes me think that they don't understand what they're talking about which is why they're using complicated terminology to mask it but I'm hopeful I'm wrong and this is an engineering innovation that benefits everyone.
There may or may not be something there, but the article is mostly buzzword-slinging. They wrote "This will allow us to put an Extropic accelerator in every home, enabling everyone to partake in the thermodynamic AI acceleration." Huh?
If they said something like "We are trying to cut the cost of stable diffusion by a factor of 100", that would sort of make sense. But then people would want to see a demo.
A proof-of-concept would be amazing and that's what I thought they were releasing and would justify the hype (as opposed to a whitepaper). Maybe in a couple of months we'll see a HN post doing exactly this and we can eat our words (which I really hope is the case).
Since they're building a special-purpose accelerator for a certain class of models, what I'd like to see is some evidence that those models can achieve competitive performance (once the hardware is mature). Namely, simulate these models on conventional hardware to determine how effective they are, then estimate what the cost would be to run the same model on Extropic's future hardware.
Much, much better. The first minute or so explains what they are trying to do and why in a way the I can understand.
This interview makes me much more excited and less skeptic than Verdon's usual mumbo-jumbo jargon. He should try using simpler, and more humble language more often.
This interview makes their product seem like BS. First, they literally cannot simply explain the problem or solution. Regardless, their pitch is that they're building a more power efficient probability distribution sampler. No one in AI research thinks that's a bottleneck.
edit: btw the bottleneck in AI algos is matrix multiply and memory bandwith.
My take on the Garry Tan interview (which seems pretty clear, regardless of whether this is snake oil or not) is that Extropic are building low power analog chips because we're hitting up against the limits of Moore's Law (limit's of physics in reducing transistor size), and at the same time the power consumption for LLM/AI training and inference is starting to get out of hand.
So, their solution is to embrace the stochastic operation of smaller chip geometries where transistors become unreliable, and double down on it by running the chips at low power where the stochasticity is even worse. They are using an analog chip design/architecture of some sort (presumably some sort of matmul equivalent?) and using a "full-stack" design whereby they have custom software to run neural nets on their chips, taking advantage of the fact the neural nets can tolerate, and utilize, randomness.
Just watched a few minutes of the Lex interview, and have to say Verdon gives off a totally different vibe there, and seems to be talking gibberish about quantum computing.
However, the idea of using analog matrix multiply is reasonable, and has already been done by at least one company:
Computationally, yes, those are the bottlenecks. But I would also add supervised training data, as we can never get enough of that and it is one of few things that increases in compute are (to my mind, you could argue that by scaling unsupervised training further we could do away with it, but I am not yet convinced) not able to solve.
Their startup is addressing computing bottlenecks so that's what I addressed. Supervised training dat isn't a bottleneck on LLMs, Diffusion models, or any of the hot areas at the moment.
I think the situation is less clear than that. While I have limited research experience with image generation, I believe I do have a fair understanding of large language models. From the publication of GPT-2 until ChatGPT, it was true that the argument always was that supervised training data was not a priority and that it all boiled down to scaling the amount of unsupervised training data. However, this all changed with preference tuning, etc. and I think there is also an argument to be made that the extensive training data curation that we see today (and is withheld from the "papers" we see for the models) is a form of supervision in its own right. It could be that we will see computational/data scaling dominate again, but I think it is equally possible that we will have the next few years dominated by data curation and exploring forms of supervision to "extract" value out of what was learnt at the unsupervised training stage.
Still, you are correct that Extropic is looking at the computation rather than data. But, I wanted to chime in so as the discussion here would not leave the impression that we are still in the days of pure unsupervised scaling.
The sad truth: get on Twitter and say a lot of weird, "high-minded" things. It's where VCs hang out, and this is the language they get from a lot of people.
People need to read Hamming’s old papers in which he very clearly explains why analog circuits are not viable at scale. This is also why the brain uses spikes rather than continuous signals. The issue is noise, interference, and attenuation. There’s no way to get around this. If they have invented a way, I’d like to see it. But until it’s demonstrated, I’d take such things with a large grain of salt.
You can re-quantize analog signals into a finite number of levels to prevent noise accumulation. That's how TLC (8 levels) and QLC (16 levels) flash memory cells work. The cells store an analog value, but it's forced to a value close to one of N discrete values. The same approach is used in modems.
Deep learning doesn't seem to need that much numerical precision. People started with 32-bit floats, then 16-bit floats, now sometimes 8-bit floats, and recently there are people talking up 2-bit trinary. The number of levels needed may not be too much for analog. If you have a regenerator once in a while to slot values back to the allowed discrete levels, you can clean up the noise. That's an analog to digital to analog conversion, of course.
That's not what these guys are talking about, as far as I can tell.
Not at the moment, but I do recall he has a chapter on this in his book “The Art of Doing Science and Engineering”, which I also recommend. He uses very long transmission lines to explain this, but the same thing applies at the nano scale, and perhaps to an even greater extent due to the much noisier environment and higher frequencies.
I really hope this was an experiment in using gen AI:
“Create a website for a new company that is building the next generation of computing hardware to power AI software. Make sure it sounds science-y but don’t be too specific.”
HN has always been a tense standoff between a few cliques, the first two being the ostensibly intended audience;
* competent and curious engineers
* entrepreneurs, who live on a continuum where one end is...
* ...hucksters and snake-oil purveyors, of which there are plenty, and
* (because this is the Internet) conspiracy theorists and other such loons
and recently
* political provocateurs
You can make a thread work (for that group of people) if it self-selects who reads it. Unfortunately, AI is catnip to all five of these groups, so the average thread quality is exceptionally low – it serves all five groups badly.
Whether some of these people _should_ be served well is a separate question.
binoct's comment is not "pointing out BS" - that requires that you, y'know, actually point out things. binoct's comment is sneering, without any content or thought. Pure fluff.
The use of “full-stack” was the first thing I noticed. Everyone, please stop using that term. I’m pretty sure, with a high degree of certainty, you don’t know what it means. If you do, there’s a merit badge waiting for you. And can we please stop using “hallucinations” to describe output. Yes, it may look like your tool dropped acid, but that’s not what it is.
Well, quantity has a quality all its own. Extending the sprint to wait for the person to complete their PhD as part of the "research" part of the ticket would not quite be Scrum.
I now think of the "stack" of a modern business as starting with physics and ending with making someone happy (unless you are Oracle). Full-stack engineers should then know how to connect physics to peoples' happiness.
At a high level it is the right answer to the data center electricity demand problem. Which is that we need to make AI hardware more efficient.
Pragmatically, it doesn't make much sense given that it would take years for this approach to have any real work use cases in a best case scenario. It seems way more likey that efficiency gains in digital chips will happen first making these chips less economically valuable.
For the longest time I thought the person behind the account was just some random guy who was probably very into crypto and decided to dabble in AI because of the parallels between e/acc and the whole "to the moon" messaging you find in crypto communities.
Never would have guessed the guy was an actual physicist
Hard time believing this is legit given how much time the CEO spends goofing around on social media. If it were possible to short startups, this would be a top candidate.
honestly, it would be too early to say this. Considering the people who invested in this startup, its better to assume CEO is capable. If he is not able to deliver in reasonable timeline then, we all are free to blame him for posting things on SM. actually many knows his company because he is goofing around on SM especially e/acc stuff.
exactly!! most of the large accounts hyping this stuff up are invested in it or dont seem to have a clear idea of what they invested in and/or part of the inner clique. also there doesn't seem to any serious hardware investors in that group - more folks who have bought into e/acc and all the scifi mythos around it.
So, basically this seems to be a way to replace PRNGs with real randomness with some knobs so you can adjust the distribution. Let's assume for the sake of argument that this can replace every single PRNG call in inference and training, how much savings in cost/energy/run time would there actually be?
Could someone smarter than me explain if this is a big deal or just hype? The work sound promising, but I wonder how long it would take to build and validate.
After skimming the article, go back to the beginning, and ponder the opening stanza:
> We are very excited to finally share more about what Extropic is building: a full-stack hardware platform to harness matter's natural fluctuations as a computational resource for Generative AI.
This is New Age, dressed up with the latest fashion.
I can sell you a solution to that in AWS/Azure (or on prem) today if you really want to use a TRNG for your ML training :)
They are very energy efficient (measured in pJ/bit), but non-cryptographic PRNGs, which are typical for ML, are far more efficient.
It's not obviously wrong to think that AI algorithms will pick up bias from "overfitting" to their PRNGs used during training, but I'm not expecting the benefits to be very large.
Sounds like a really convoluted new agey way of talking about some kind of analog computing.
AFAIK there are other efforts to develop analog neural network ASICs. Since neural networks are noise-tolerant this could work and could allow faster computations than conventional must-be-perfect digital circuits. IBM, Intel, and others have experimented with this.
I wouldn't believe there's anything particularly novel here unless a lot more detail or test hardware is given.
I'm not 100% sure this is true but I've heard that this fellow was involved with the NFT craze and made money there, and that sets off alarm bells. I've suspected for a while that e/acc is a marketing thing since it's just repackaging old extropian stuff from the 1990s.
"I want to believe" but have seen enough to be skeptical of extreme claims without hard evidence.
on the other hand, HN is filled with self-proclaimed critics that they dismiss everything and display their utter lack of imagination -- like AI, Metaverse, (success of) Snapchat, AirPods before
Did HN dismiss AI? I saw some skepticism and still do but not dismissal.
This site doesn’t get everything right. It tends to miss things that succeed in the consumer space because this is a pro audience not a mainstream audience. But it usually gets hard science right.
I don't know about the AI aspect, but this sounds perhaps related to probabilistic finance simulations (Black-Scholes, Heston, etc). I've heard rumors that these types of simulations account for an obscene amount of compute at AWS
I'm similarly suspicious, and find it curious that this is the first I'm hearing about this at all. I don't have personal connections in physics or AI circles but I feel like I'd usually expect to have read mention of these ideas before finding this press release.
My takeaway is that the chip's goal is to provide a way to produce random numbers with some configurable distribution that is faster and more energy efficient.
As far as the feasibility and impact on AI in general, I have no idea.
It's a startup with well-credential and very technical founders and a fair seed round focused on accelerating one bottleneck in a newly popular computing paradigm using techniques that are known in research but never yet commercialized.
It might fail for the reasons many startups fail, but it's not prima facie fantasy.
They're not wrong that sampling a complex, higher-dimensional probability distribution is hard to do efficiently. I'm not sure how useful it is to do it more efficiently, though.
Also, the fact that they're using ultra-cold superconductors makes me wonder how much noise helps and how much it hurts. If your system is all about leveraging noise well, but you can only use super special well-behaved noise, then "bad noise" could easily ruin the quality of your generated solutions.
It's cool to see something so wacky out there, though!
Honestly, as interesting as the the chip sounds, I'm admittedly kind of biased against the company's probability of success simply because the founder is basically the #1 e/acc meme account/shitposter on Twitter.
Like, it's hard to take someone seriously when they spend tons of time shitposting on Twitter, it's even harder when it's revealed that they're behind one of the most popular shitposting accounts within a niche, almost cult-like community.
Back in the day, there was a saying that went something like: Steve Jobs was really good at what he did, and also an asshole to people. The former is really hard to replicate, so instead you'll find a lot of people going around and imitating the latter.
Today, it's a very different situation. Now we have Elon Musk, who is really good at what he does, and also tweets a lot...
To be fair it isn't very common to detail proprietary hardware in github repos. And any code for such novel processors would be fascinating but useful only for theory rather than practice at the moment. The lack of open code is a missing merit badge rather than a demerit.
Physical learning machines require noise to learn. They are also necessarily dissipative. See https://arxiv.org/abs/2209.11954. The key is to engineer the noise to maximise the learning rate. In classical devices, stochastic switching is controlled by temperature through the Kramers rate. This means kT controls energy loss. If you use dissipative quantum tunnelling this is not the true thermodynamic lower bound. Any quantum nonlinear dissipative system, with a far from equilibrium steady state, is a good case to consider. Dispersive optical bistability, realised in SC quantum circuits, is the way to go. And quantum error correction is unnecessary.
The amount of buzzwords on this page should disqualify this from even getting votes on HN. Anyone who writes like this is trying to confuse and mislead the reader.
Too early to tell about what this will be in the future. Either it turns out to be a foundational startup or a flash in a pan.
But at least it is not the 5000th so-called AI-powered SaaS company that is using OpenAI API that has raised $20M+ to VCs and burning hundreds of thousands every month with little to no plan to generate revenue.
Will be watching this one closely, but highly skeptical of this company.
Hear hear, better to see someone go for broke trying something novel.
At best they advance the field massively, at worst the backers lose their money but the tech/knowledge finds a home elsewhere and the knowledge in the field is nudged forward.
Man, I am not a pessimist and I am very bullish on AI-the-field but my spidey sense is tingling that this is BS.
- It is written in a way that sacrifices legibility for supposed precision but because the terms used can't really be applied precisely, it's equivalent to spurious digits in a scientific calculation. The usual reason this occurs is to obfuscate or to overawe the audience.
- It is hard to overstate the difficulty of beating semiconductor with a wholly new branch of technology. They're so insanely good. People have been trying to beat them for decades and there's not even a solid theoretical thesis as to how to do so. Even the theoretical advantage of quantum computing is predicated on error correction being scalable which is a totally open question even theoretically.
If room-temperature-stable bio-enhanced AI-specific-computer-powered chatbots don't seem like a realistic goal then maybe you should have clicked "play" on the linked spotify widget.
For me its the dichotomy between how absolutely impenetrable the blog post is, combined with the "Set the tone fam, play 'Entropy' by Noizinski on Spotify :)" widget in the bottom right. Like they're trying to check every box on the engagement farming list (something, to be sure, beff jezos is famous for).
Very bad vibes. Hire someone who can communicate, and demonstrate what you're building.
It feels like serious people would have said something more like "we are going to improve the performance (measured in s), of the algorithms/models such X, Y, Z which are used in a, b, c."
Can anyone name a company which used such absurd language to describe themselves and then actually delivered something valuable? There must be one.
I'm saddened to see the honorable name of Extropy and Extropianism, which carefully never descended to this level or anything like it, be stolen and captured by this nonsense.
No, not sarcasm, and I am Eliezer Yudkowsky. I was around on the old Extropians mailing list starting in 1996, and their leadership did not talk like this. Max More (the founder of Extropianism) was a careful thinker then, and I haven't heard anything different about him more recently than that.
"Extropy" is a term that was previously coined by a group of fairly nice people to describe themselves, and so far as I know is being stolen here without permission.
The word "extropy" itself seems to go back several decades before the mailing list, if I'm reading correctly here: https://en.wikipedia.org/wiki/Extropianism. Still, I wouldn't be surprised if many/most of the original mailing list members found this usage a corruption.
I've thought for a while that what quantum computing will probably deliver is not going to be magical infinite processing power, but extremely fast, computational access to parameterizable physical processes. That is, a rock can simulate being a rock better than a computer can, but how do you hook it up to the rest of your system? But while I can imagine replacing a simple MCMC model, for example, with a stack of physics-based chips, is there a path all the way to designing, training and executing something LLM sized on top of that technology? I'm not smart enough to know, but as esoteric as it sounds, it feels like it's drawing on the less speculative end of the spectrum, and seems like a noble effort and not an actual scam.
It's a random, unassuming 7-year-old blog post from a DARPA-funded and defense-involved inventor. They happen to work in neuromorphic computing. Their other posts talk about some of that work. A cynical take is that it can seem like just hand-wavey garbage, but then again, it's been quietly getting tons of defense contractor money.
I came across it years ago, and it has greatly accelerated my worldview, and has made me feel ahead of the curve in understanding what is going on in the universe. It's informed my community organizing. It's informed how I understand AI and consciousness and language, and the intersection of all these things.
I'm inclined to believe that the people in this area are clued into something very substantial about how the universe works.
EDIT: oops, shared the wrong link. This one is about thermodynamic evolution
Seems like they're "passive" energy chips are only gonna be targeted $$$ towards big organizations, which make use of the Josephon effect. But if they're targeting transistor technology for the masses, how will they have an advantage against the incumbents
for all the hype around building alien tech, this is a bit underwhelming. the stuff from this startup feels more alien than what extropic is talking about - https://www.emergentia.tech/technology
Not only does this read like pure bullshit, it is bullshit on a website that crashes the Apple Vision Pro (and makes my laptop suffer).
My prediction is that they will raise a nine-figure sum over the next decade, and never release a product that comes close to the performance of an NVIDIA card today.
I know everyone is calling BS on this, and I am just a simple web developer so what do I know but there are at least two priors that make me think that what is discussed here could have some validity.
* The stochastic/random nature of processors is already used in cryptography for physically uncloneable functions. Dunno if this has any practical uses in industry, and it is crypto, so it is probably also BS, but it is the same phenomena you get if you log in into your BIOS and turn off ECC of your RAM.
* The very first computer capable of MCMC was designed by von Neumann himself and used uranium as a source of randomness as part of the Manhattan project.
Anyway semiconductors have never been my strong suit, but I guess this is more of a IP play then a consumer product business. Now let me get back to writing unit tests.
The litepaper discusses Extropic's mission to develop a novel hardware platform that harnesses the natural fluctuations of matter as a computational resource for Generative AI.
Key Points
The demand for computing power in AI is increasing exponentially, but Moore's Law is slowing down due to fundamental physical limitations of transistors at the atomic scale.
Biology hosts more efficient computing circuitry than current human-made devices by leveraging intrinsic randomness in chemical reaction networks.
Energy-Based Models (EBMs) are a potential solution, as they are optimal for modeling probability distributions and require minimal data. However, sampling from EBMs is difficult on digital hardware.
Extropic is implementing EBMs directly as parameterized stochastic analog circuits, which can achieve orders of magnitude improvement in runtime and energy efficiency compared to digital computers.
Extropic's first processors are nano-fabricated from aluminum and run at low temperatures where they are superconducting, using Josephson junctions for nonlinearity.
Extropic is also developing semiconductor devices that operate at room temperature, sacrificing some energy efficiency for scalability and accessibility.
A software layer is being built to compile abstract specifications of EBMs to the relevant hardware control language, enabling Extropic accelerators to run large programs.
They’re not building “a better rng”- they’re building a way to bake probabilistic models into hardware and then run inference on them using random fluctuations. Theoretically this means much faster inference for things like PGMs.
See here for similar things: https://arxiv.org/abs/2108.09836
There’s a company called Normal Computing that did something similar: https://blog.normalcomputing.ai/posts/2023-11-09-thermodynam...