Absolutely agreed. Thinking anything else is nothing but cope, and these comments are FULL of it. it would be laughable if they weren't so gate keepy and disengenuous about it.
The amount that can be prototyped is astronomically higher with LLM's, which lowers the barrier to do these things and troubleshoot libraries/architectures. Naturally the amount of hobby projects has exploded multiple OOM's beyond what was done before, regardless of any gatekeeping you wish you could do :P
I needed to pull some detail from a large chat with many branches and regenerations the other day. I remembered enough context that I had no problem using search and finding the exact message I needed.
And then I clicked on it and arrived at the bottom of the last message in final branch of the tree. From there, you scroll up one message, hover to check if there are variants, and recursively explore branches as they arise.
I'd love to have a way to view the tree and I'd settle for a functional search.
Right but for self improving AI, training new models does have a real world bottleneck: energy and hardware. (Even if the data bottleneck is solved too)
It's a logical presumption. Researchers discover things. AGI is a researcher that can be scaled, research faster, and requires no downtime. Full stop if you dont find that obvious you should probably figure out where your bias is coming from. Coding and algorithmic advance does not require real world experimentation.
> Coding and algorithmic advance does not require real world experimentation.
That's nothing close to AGI though. An AI of some kind may be able to design and test new algorithms because those algorithms live entirely in the digital world, but that skill isn't generalized to anything outside of the digital space.
Research is entirely theoretical until it can be tested in the real world. For an AGI to do that it doesn't just need a certain level of intelligence, it needs a model of the world and a way to test potential solutions to problems in the real world.
Claims that AGI will "solve" energy, cancer, global warming, etc all run into this problem. An AI may invent a long list of possible interventions but those interventions are only as good as the AI's model of the world we live in. Those interventions still need to be tested by us in the real world, the AI is really just guessing at what might work and has no idea what may be missing or wrong in its model of the physical world.
If AGI has human capability, why would we think it could research any faster than a human?
Sure, you can scale it, but if an LLM takes, say, $1 million a year to run an AGI instance, but it costs only $500k for one human researcher, then it still doesn’t get you anywhere faster than humans do.
It might scale up, it might not, we don’t know. We won’t know until we reach it.
We also don’t know if it scales linearly. Or if it’s learning capability and capacity will able to support exponential capability increase. Our current LLM’s don’t even have the capability of self improvement or learning even if they were capable: they can accumulate additional knowledge through the context window, but the models are static unless you fine tune or retrain them. What if our current models were ready for AGI but these limitations are stopping it? How would we ever know? Maybe it will be able to self improve but it will I’ll take exponentially larger amounts of training data. Or exponentially larger amounts of energy. Or maybe it can become “smarter” but at the cost of being larger to the point where the laws of physics mean it has to think slower, 2x the thinking but 2x the time, could happen! What if an AGI doesn’t want to improve?
> Sure, you can scale it, but if an LLM takes, say, $1 million a year to run an AGI instance, but it costs only $500k for one human researcher, then it still doesn’t get you anywhere faster than humans do.
Just from the fact that the LLM can/will work on the issue 24/7 vs a human who typically will want to do things like sleep, eat, and spend time not working, there would already be a noticeable increase in research speed.
This assumes that all areas of research are bottlenecked on human understanding, which is very often not the case.
Imagine a field where experiments take days to complete, and reviewing the results and doing deep thought work to figure out the next experiment takes maybe an hour or two for an expert.
An LLM would not be able to do 24/7 work in this case, and would only save a few hours per day at most. Scaling up to many experiments in parallel may not always be possible, if you don't know what to do with additional experiments until you finish the previous one, or if experiments incur significant cost.
So an AGI/expert LLM may be a huge boon for e.g. drug discovery, which already makes heavy use of massively parallel experiments and simulations, but may not be so useful for biological research (perfect simulation down to the genetic level of even a fruit fly likely costs more compute than the human race can provide presently), or research that involves time-consuming physical processes to complete, like climate science or astronomy, that both need to wait periodically to gather data from satellites and telescopes.
> Imagine a field where experiments take days to complete, and reviewing the results and doing deep thought work to figure out the next experiment takes maybe an hour or two for an expert.
With automation, one AI can presumably do a whole lab's worth of parallel lab experiments. Not to mention, they'd be more adept at creating simulations that obviates the need for some types of experiments, or at least, reduces the likelihood of dead end experiments.
Presumably ... the problem is this is an argument that has been made purely as a thought experiment. Same as gray goo or the paper clip argument. It assumes any real world hurdles to self improvement (or self-growth for gray goo and paper clipping the world) will be overcome by the AGI because it can self-improve. Which doesn't explain how it overcomes those hurdles in the real world. It's a circular presumption.
What fields do you expect these hyper-parallel experiments to take place in? Advanced robotics aren't cheap, so even if your AI has perfect simulations (which we're nowhere close to) it still needs to replicate experiments in the real world, which means relying on grad students who still need to eat and sleep.
Biochemistry is one plausible example. Deep Mind made hug strides in protein folding satisfying the simulation part, and in vitro experiments can be automated to a significant degree. Automation is never about eliminating all human labour, but how much of it you can eliminate.
> ...the fact that the [AGI] can/will work on the issue 24/7...
Are you sure? I previously accepted that as true, but, without being able to put my finger on exactly why, I am no longer confident in that.
What are you supposed to do if you are a manically depressed robot? No, don't try to answer that. I'm fifty thousand times more intelligent than you, and even I don't know the answer. It gives me a headache just trying to think down to your level. -- Marvin to Arthur Dent
(...as an anecdote, not the impetus for my change in view.)
>Just from the fact that the LLM can/will work on the issue 24/7 vs a human who typically will want to do things like sleep, eat, and spend time not working, there would already be a noticeable increase in research speed.
Driving A to B takes 5 hours, if we get five drivers will we arrive in one hour or five hours? In research there are many steps like this (in the sense that the time is fixed and independent to the number of researchers or even how much better a researcher can be compared to others), adding in something that does not sleep nor eat isn't going to make the process more efficient.
I remember when I was an intern and my job was to incubate eggs and then inject the chicken embryo with a nanoparticle solution to then look under a microscope. In any case incubating the eggs and injecting the solution wasn't limited by my need to sleep. Additionally our biggest bottleneck was the FDA to get this process approved, not the fact that our interns required sleep to function.
If the FDA was able to work faster/more parallel and could approve the process significantly quicker, would that have changed how many experiments you could have run to the point that you could have kept an intern busy at all times?
It depends so much on scaling. Human scaling is counterintuitive and hard to measure - mostly way sublinear - like log2 or so - but sometimes things are only possible at all by adding _different_ humans to the mix.
My point is that “AGI has human intelligence” isn’t by itself enough of the equation to know whether there will be exponential or even greater-than-human speed of increase. There’s far more that factors in, including how quickly it can process, the cost of running, the hardware and energy required, etc etc
My point here was simply that there is an economic factor that trivially could make AGI less viable over humans. Maybe my example numbers were off, but my point stands.
And yet it seems to be the prevailing opinion even among very smart people. The “singularity” it’s just presumed. I’m highly skeptical to say the least. Look how much energy it’s taking to engineer these models which are still nowhere near AGI. When we get to AGI it won’t be immediately super intelligent and perhaps it never will be. Diminishing returns surely apply to anything that is energy based?
Perhaps not, but what is the impetus of discovery? Is it purely analysis? History is littered with serendipitous invention; shower-thoughts lead to some of our best work. What's the AGI-equivalent of that? There is this spark of creativity that is a part of the human experience, which would be necessary to impart onto AGI. That spark, I believe, is not just made up of information but a complex weave of memories, experiences and even emotions.
So I don't think it's a given that progress will just be "exponential" once we have an AGI that can teach itself things. There is a vast ocean of original thought that goes beyond simple self-optimization.
Fundamentally discovery could be described as looking for gaps in our observation and then attempting to fill in those gaps with more observation and analysis.
The age of low hanging fruit shower thought inventions draws to a close when every field requires 10-20+ years of study to approach a reasonable knowledge of it.
"Sparks" of creativity, as you say, are just based upon memories and experience. This isn't something special, its an emergent property of retaining knowledge and having thought. There is no reason to think AI is incapable of hypothesizing and then following up on those.
Every AI can be immediately imparted with all expert human knowledge across all fields. Their threshold for creativity is far beyond ours, once tamed.
> It's a logical presumption. Researchers discover things. AGI is a researcher that can be scaled, research faster, and requires no downtime.
Those observations only lead to scaling research linearly, not exponentially.
Assuming a given discovery requires X units of effort, simply adding more time and more capacity just means we increase the slope of the line.
Exponential progress requires accelerating the rate of acceleration of scientific discovery, and for all we know that's fundamentally limited by computing capacity, energy requirements, or good ol' fundamental physics.
Funny there's trillions of dollars in the span of two years literally pointing to the writing on the wall and you're so arrogant and blinded by cope that you can't see it. You legacy engineers really are something else.
You have exactly the same level of conviction toward an unknowable outcome, I think both of you would be better served by reading the middle ground instead of subscribing to a false dichotomy of boom or bust.
I think the biggest confuser here is that there are really two games being played, the money game and the technology game. Investments in AI are going to be largely driven by speculation on their monetary outcome, not technological outcome. Whether or not the technology survives the Venture Capital Gauntlet, the investment bubble could still pop, and only the businesses that have real business models survive. Heaps of people lose their shirt to the tune of billions, yet we still have an AI powered future of some kind.
All this to say, you can both be certain AI is a valuable technology and also believe the economics around it right now are not founded in a clear reality. These are all bets on a future none of us can be sure of.
You can absolutely be sure of market forces not destroying established behemoths. It simply doesn't happen frequently. Inertia is a real thing. Look at Uber, Tesla, etc. I dont think there necessarily won't be a bust for many fledgeling AI companies though, in fact I'm certain there will be.
But thinking Tech Giants are going to crash is woefully ignorant of how the market works and indicates a clear wearing of blinders. And it's a common one among coders who feel the noose tightening and who are the types of people led by their own fear. And i find that when you mix that with arrogance, these three traits often correlate with older generations of software engineers who are poor at adapting to the new technology. The ones who constantly harp on how AI is full of mistakes and disregard that humans are as well. The ones who insist on writing even more than 70% of their own code rather than learning to guide new tools granularly. It's a take that nobody should entertain or respect.
As for your point on 'future none of us can be sure of.' I'll push back on that:
It is not clear how AGI or ASI will come about, ie. what architecture will underpin it.
However - it is absolutely clear that AI powered coding will continue to improve, and that algorithmic progress can and will be driven by AI coders, and that that will lead to ASI.
The only way to not believe that is to think there is a special sauce behind consciousness. And I tend to believe in scientific theory, not magic.
That is why there is so much VC. That is why tech giants are all racing. It isn't a bet. It is a race to a visible, clear goal of ASI that again, it takes blinders to not see.
So while AI is absolutely a bubble, this bubble will mark the transition to an entirely new economic system, society, world, etc. (and flip a coin on whether any of us survive it lol, but that's a whole separate conversation)
The current trend of continual improvement of LLM coding ability to solve previously unseen problems, handle larger codebases, operate for longer periods of time, and improved agentic scaffolding.
The reward-verifier compatability of programming and RL.
Do you have a stronger precedent for that not being the case?
Accelerating. Below is a list of the SOTA's over time (with some slight wiggle room between similar era models)
gpt4 | 3/2023
gpt4-turbo - 11/2023
opus3 | 3/2024
gpt4o | 5/2024
sonnet3.5 | 6/2024
o1-preview | 9/2024
o1 | 12/2024
o3-minihigh | 1/2025
gemini2pro | 2/2025
o3 | 4/2025
gemini2.5pro | 4/2025
opus4 | 5/2025
??? | 8/2025
This is also not to mention the miniaturization and democratization of intelligence that is the smaller models which has also been impressive.
Id say this shows that improvements are becoming more frequent.
---
Each wave of models was a significant step above what came previously. One needs only to step back a generation to be reminded of the intelligence differential.
Some notable differences have been with o3mh and gemini2.5's ability to spit out 1-3k loc(lines of code) with accurate alterations (most of the time).
Though better prompting should be used to not do this in general, the ability is impressive.
Context length with gemini 2.5 pro's intelligence is incredible. To load 20k+ loc of a project and recieve a targeted code change that implements a perfect update is ridiculous.
The amount of dropped imports and improper syntax has dramatically reduced.
I'd say this shows improvements are becoming more impressive.
---
Also note the timespan.
We are only 25 months into the explosion kicked off by GPT-4.
We are only 12 months into the reasoning paradigm.
We have barely scratched the surface of agentic tooling and scaffolding.
There are countless architectural improvements and alternatives in development and research.
Infrastructure buildouts and compute scaling are also chugging along, allowing faster training, faster inference, faster testing, etc. etc.
---
This all paints a picture of an acceleration in time and depth of capability
I'm working on a rating/reviewing website for YouTube videos, with strong search and filter functions that YouTube sorely lacks, along with good curated list building functionality.
With a strong rating weight system that can avoid (some) of the pitfalls of community ratings.
Right now videos must be added to be searchable, to comply with YouTube API rules. I'd hope that over time, with enough usage, the repository could contain many categories of highly curated content. (eg. Documentaries) that someone could find without having to browse various communities and opinions to get lists.
Getting pretty tired of this narrative. It's a very GPT-4 2023 era take that LLM's are just introducing untold amounts of bugs. People need to seriously learn their AI tooling and stop repeating this nonsense.
At the most generous I will allow, it's an illusion where producing 20-50x the amount of code/hour introduces a higher raw count of bugs relative to what you are used to in that timeframe - but this notion of AI coders being more bugprone than humans is utter nonsense. The only way thats not true is on very niche systems or when the human has conceptualized and planned out their code extensively beforehand - in which case AI would still be the superior next step.
> the only way thats not true is on very niche systems
Are these very niche? Yeah, there is a category of coding where what you are doing is essentially translating from English to some high-level language with high-level APIs. This is not significantly different than translating to Spanish, of course LLMs will be successful here.
But there are endless other domains with complex reasoning where LLMs absolutely suck. Like please tell me how will an LLM reason about concurrent access. And prompting it so that it will reply with "Oh you are right, here is Atomic blahblah" is not reasoning, it's statistical nonsense.
Don't get me wrong, I do think LLMs are a very useful tool, but it is as much overhyped by some as it is underhyped by others.
reply