I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
It'd be naive if it wasn't literally a standard point that is addressed and acknowledged as being a major part of the problem.
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
How does that address the issue? I would have expected them to do that anyhow. Thats what a lot of businesses do: let another company take the hit developing the market, R and D, and supply chain, then come in with industry standardization and cooperative agreements only after the money was proven to be good in this space. See electric cars. Also they could drop that at any time. Remember when openAI stood for opensource?
Neither mention anything about open-source, although a later update mentions publishing work (“whether as papers, blog posts, or code”), which isn't exactly a ringing endorsement of “everything will be open-source” as a fundamental principle of the organization.
Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.
Assuming AGI works like a braindead consulting firm, maybe. But if it worked like existing statistical tooling (which it does, today, because for an actual data scientist and not aunt cathy prompting bing, using ml is no different than using any other statistics when you are writing your python or R scripts up), you could probably generate some fancy charts that show some distributions of cars produced under different scenarios with fixed resource or power limits.
In a sense this is what is already done and why ai hasn't really made the inroads people think it will even if you can ask google questions now. For the data scientists, the black magicians of the ai age, this spell is no more powerful than other spells, many of which (including ml) were created by powerful magicians from the early 1900s.
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.
There's a reason OpenAI had this as part of its charter:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
This goes massively against the consensus of experts in this field. The modal AI researcher believes that "high-level machine intelligence", roughly AGI, will be achieved by 2047, per the survey below. Given the rapid pace of development in this field, it's likely that timelines would be shorter if this were asked today.
I am in the field. The consensus is made up by a few loudmouths. No serious front line researcher I know believes we’re anywhere near AGI, or will be in the foreseeable future.
So the researchers at Deepmind, OpenAI, Anthropic, etc, are not "serious front line researchers"? Seems like a claim that is trivially falsified by just looking at what the staff at leading orgs believe.
Apparently not. Or maybe they are heavily incentivized by the hype cycle. I'll repeat one more time: none of the currently known approaches are going to get us to AGI. Some may end up being useful for it, but large chunks of what we think is needed (cognition, world model, ability to learn concepts from massive amounts of multimodal, primarily visual, and almost entirely unlabeled, input) are currently either nascent or missing entirely. Yann LeCun wrote a paper about this a couple of years ago, you should read it: https://openreview.net/pdf?id=BZ5a1r-kVsf. The state of the art has not changed since then.
I don't give much credit to the claim that it's impossible for current approaches to get us to any specific type or level of capabilities. We're doing program search over a very wide space of programs; what that can result in is an empirical question about both the space of possible programs and the training procedure (including the data distribution). Unfortunately it's one where we don't have a good way of making advance predictions, rather than "try it and find out".
It is in moments like these that I wish I wasn’t anonymous on here and could bet a 6 figure sum on AGI not happening in then next 10 years, which is how I define “foreseeable future”.
You disagreed that 2047 was reasonable on the basis that researchers didn't think it wouldn't happen in the foreseeable future, so your definition must be at least 23 years for consistency's sake
I'd be OK with that, too, if we adjusted the bet for inflation. This is, in a way, similar to fusion. We're at a point where we managed to ignite plasma for a few milliseconds. Predictions of when we're going to be able to generate energy have become a running joke. The same will be the case with AGI.
LeCun has his own interests at heart, works for one of the most soulless corporations I know of, and devotes a significant amount of every paper he writes to citing himself.
Fair, ad hominems are indeed not very convincing. Though I do think everyone should read his papers through a lens of "having a very high h-index seems to be a driving force behind this man".
Moving on, my main issue is that it is mostly speculation, as all such papers will be. We do not understand how intelligence works in humans and animals, and most of this paper is an attempt to pretend otherwise. We certainly don't know where the exact divide between humans and animals is and what causes it, which I think is hugely important to developing AGI.
As a concrete example, in the first few paragraphs he makes a point about how a human can learn to drive in ~20 hours, but ML models can't drive at that level after countless hours of training. First you need to take that at face value, which I am not sure you should. From what I have seen, the latest versions of Tesla FSD are indeed better at driving than many people who have only driven for 20 hours.
Even if we give him that one though, LeCun then immediately postulates this is because humans and animals have "world models". And that's true. Humans and animals do have world models, as far as we can tell. But the example he just used is a task that only humans can do, right? So the distinguishing factor is not "having a world model", because I'm not going to let a monkey drive my car even after 10,000 hours of training.
Then he proceeds to talk about how perception in humans is very sophisticated and this in part is what gives rise to said world model. However he doesn't stop to think "hey, maybe this sophisticated perception is the difference, not the fundamental world model". e.g. maybe Tesla FSD would be pretty good if it had access to taste, touch, sight, sound, smell, incredibly high definition cameras, etc. Maybe the reason it takes FSD countless training hours is because all it has are shitty cameras (relative to human vision and all our other senses). Maybe linear improvements in perception leads to exponential improvement in learning rates.
Basically he puts forward his idea, which is hard to substantiate given we don't actually understand the source of human-level intelligence, and doesn't really want to genuinely explore (i.e. steelman) alternate ideas much.
Anyway that's how I feel about the first third of the paper, which is all I've read so far. Will read the rest on my lunch break. Hopefully he invalidates the points I just made in the latter 2/3rds.
This could also just be an indication (and I think this is the case) that many Manifold betters believe the ARC AGI Grand Prize to be not a great test of AGI and that it can be solved with something less capable than AGI.
I don't understand how you got 2047. For the 2022 survey:
- "How many years until you expect: - a 90% probability of HLMI existing?"
mode: 100 years
median: 64 years
- "How likely is it that HLMI exists: - in 40 years?"
mode: 50%
median: 45%
And from the summary of results: "The aggregate forecast time to a 50% chance of HLMI was 37 years, i.e. 2059"
That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.
Not sure if you want to share the capitalist system with an entity that outcompetes you by definition. Chimps don't seem to do too well under capitalism.
You might be right, but that wasn't my point. Capitalism might yield a friendly AGI or an unfriendly AGI or some mix of both. Collectivism will yield no AGI.
One can already see the beginning of AI enslaving humanity through the establishment. Companies work on AI get more investment and those who don't gets kicked out of the game. Those who employ AI get more investment and those who pay humans lose confidence through the market. People lose jobs, get harshly low birth rates while AI thrives. Tragic.
So far it is only people telling AI what to do. When we reach the day where it is common place for AI to tell people what to do then we are possibly in trouble.
It is a trendy but dumbass tautology used by intellectually lazy people who think they are smart. Society is based upon capitalism therefore everything bad is the fault of capitalism.
