> Corporations may be viewed as special kinds of artificial intelligences whose building blocks (humans) are cogs in the machine (who for the most part may not always perceive the consequences of the corporation’s overall behavior)
> Now imagine AI systems like corporations that (a) could be even smarter than our largest corporations and (b) can run without humans to perform their actions (or without humans understanding how their actions could contribute to a nefarious outcome)
Wait - but it’s those very corporations who currently control the most powerful AI on the planet - so in a sense this is an argument that we already have rogue AI. Corporations with goals that are not aligned with society, not controlled by any individual and outside the effective control of government entities, who now provide essentially unlimited compute and human researcher/programmer resources to improve the AI. Who’s in control? Nobody.
This is so on par with Frank Herbert's prophecy from Dune: "Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."
Bengio mentions that open source is playing a leading role in AI advancements these days. So maybe it's more accurate to view open source as the corporation at the forefront. I mention this only because it's interesting to contemplate the corporate motivations of the open source community versus a traditional for-profit corporation. I know it will be a stretch for some people to imagine open source as a corporation but if you stick to Bengio's definiton ("building blocks (humans) are cogs in the machine who for the most part may not always perceive the consequences of the corporation’s overall behavior") then it'll be easier to see what I'm getting at.
Given that most major (in terms of usage) open source products are little more than commercial collaboration between large corporations (e.g. the Linux kernel, clang, Kubernetes, the various Python ML libraries) I'd say that much of the open source community is quite directly made up of corporations.
I'd say it doesn't have to be binary. Corporations are evolving to use less and less human labor. That doesn't need to be 0 for very bad outcomes for society and the human species. the stock market is already a paperclip optimizer. Most wealthy spend the bulk of their wealth in service of more paperclips plus maybe a guilt tax.
> Most wealthy spend the bulk of their wealth in service of more paperclips
What? No. They spend their money fighting with other wealthy people over who gets the passive income from the paperclip machine. The game they play is about maximizing this income. Whether or not this also maximizes paperclips is completely immaterial to them. If they had an opportunity to form a paperclip cartel and profit from cutting supply, they would be happy to do that instead. Yes, the idea of a paperclip cartel is ridiculous -- but no less ridiculous than the idea that maximizing paperclips maximizes investor profit.
Companies are about investor profit. Everything else is a side effect.
Actually ChatGPT-4 explains it way better than I do:
The author is trying to convey that the situation with corporations and human labor doesn't have to be an extreme of either using human labor or not using it at all. They argue that corporations are gradually relying less on human labor, but that doesn't mean it has to drop to zero for negative consequences to occur within society.
They also bring up the stock market as an example of a system that focuses on optimizing for profit (paperclips) rather than human welfare. The author implies that many wealthy individuals primarily focus on accumulating more wealth (paperclips) and may only contribute a small portion (guilt tax) to alleviate any sense of responsibility for the negative consequences of their actions on society.
I like the idea of “paperclips” as a substitute for the change of what money / profits mean “post Realwirtschaft”. Finance keeps on reigning supreme (ever increasingly) since the 80s. Why not change the vocabulary to better reflect that.
Respectfully, I don't think that you understand my point.
Paperclips are an extremely poor metaphor for financial assets. Rich people want to get paid for being rich. That's what financial assets do. Paperclips do not do that.
Sort of. Less cynically, the economy is not a zero sum game so some of those returns can be organic shares of positive-sum activities that make life better for everyone. Cynically, they sure don't have to be, because peasant juice tastes just as sweet. There is an awful lot of runaway rent seeking hiding in the clothes of "investment." Either way, assets represent an entitlement to the proceeds.
My point here isn't so much to pass moral judgement on capitalism as it is to assert that money and assets have fundamentally different character. They are easy to confuse due to the liquid market for exchanging the two, but the lines of reasoning that lead us to a concept of "enough money" do not lead us to a concept of "enough assets." We might independently arrive at a concept of "enough assets" by hypothesizing a situation where one person controls a 10% of world GDP or something but world GDP is 100T and the richest people own about 100B in assets so at a 10x multiple they are probably at least three orders of magnitude removed from a self-serving definition of "enough assets" (and that would be the relevant definition, since they would be the ones defining "enough.")
tl;dr Rich people DO stop maximizing money -- but then they start maximizing power, which can be bought with money, and which we measure in units of money.
Bengio is a certified and maximally informed genius, so my thoughts obviously deserve far more salt than his. But...
The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI. He goes as far as calling out open source development of AI as a contributing threat:
> [if] the open-source community continues to play a leading role in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models...
The concern is logical, but the solution is self-contradictory (as many intellectually honest explorations turn out): a section later, we are appropriately exploring nefarious corporations as examples of mis-aligned intelligences -- and how larger corporations may be even more adept at circumventing the rules.
Whatever their philosophical foundations, our governments are also corporations (and so will be any new trans-national organization). They are fragmented into smaller groups with their own instrumental goals, and subject to the same risks of corruption that afflict private corporations, and the same risks of runaway optimization that may afflict AI. Appealing to them as a higher authority is at best a case of "the devil you know".
Doomism always has a non-falsifiable counterargument: AI is new and might become arbitrarily more powerful than the most powerful organizations we've ever created ("super"-intelligence). For me this fails on several counts: we are still exploring the limits of power (and of corruption) that large organizations can achieve, progress in AI remains observable as long as it is public, and our primary experience with AI has been colonizing a frontier of capabilities rather than a sequence of new systems dominating at every task.
To be clear, I do NOT personally conclude that we should avoid all regulation and oversight. But I think that constructing regulation around doomist theories serves disproportionately towards regulatory capture, and acts as a distraction from immediate issues of algorithmic inequality, disruption of political speech, and intentional weaponization of AI.
Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.
If AI is a weapon (possible), open source is how the populous maintains freedom against corporations or governments. It’s unclear how the public keeps up with the arms war when privately trained models can have access to a wide collection of data that’s not publicly available.
AI is not a weapon. AI certainly, like every other information technology ever certainly has applications to weapons and warfare. It also has an enormous array of non-weapons applications.
If you read the GPT-4 technical report you can see that the unrestricted model (that people could be running at home) are capable of some pretty large-scale horribleness.
> The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI.
Ridiculous take. AI ain't nukes, you don't need a thousand centrifuges to separate it from raw AI ore. A huge amount of tiny actors are sufficiently equipped to work on it.
GPT-3 was trained on 1024 GPUs, which is hardly a blip on NVIDIA sales radar. Tuning an existing model, if you can get your hands on it is probably a lot less demanding.
Totally agree. Most particularly with your last point:
> Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.
AI is out and about. You cannot prevent humans from developping it.
The fear that's present in the analogy is that AIs are dangerous because they will be able to do capitalism 1000x better than corporations. That alone should make us stop and pause for reflection.
> so in a sense this is an argument that we already have rogue AI
Yes. Science fiction is pretty much 100% about things that have already happened.
Colonization by aliens? Colonization by Europeans.
Robot slave uprisings? Human slave uprisings.
AIs taking over the world? Capitalism taking over the world.
Once you realize that sci-fi has to be about things that have already happened for writers to be able to write convincingly about it, it's hard to unsee. It's all just "what if the past but in the future???"
Once they align into monopolistic cartels and integrate with government, that all changes - and that process (which took place for example in Germany in the 1930s) is well underway again. In such situations, corporations can direct governments to deploy their military forces for their own benefit.
IG Farben and a cohort of allied corporations provided critical funding to the Nazi party in the early 1930s, without which it would have had little ability to gain political power, and IG Farben's immediate reward was guaranteed government contracts for fuel, and later profited as it moved in to take over mines and factories behind the advance of the German armies.
> "By the end of 1933, the Reich agreed to buy all of the Leuna factory's output that couldn't be sold on the market, Jeffreys says. That same year, Farben donated more than 4.5 million Reich marks to Nazi Party funds." (Diarmuid Jeffreys, "Hell's Cartel", 2008)"
That is again a misunderstanding between funding/economics and actual power wielded. The Nazis had no problem flat out telling corporations and their owners what to do or not to do. The lines gets blurry if a high ranking Nazi owned a corporation or something.
Corporations might have had good business etc., but that again isn't the same as being in control.
> "Business Collaboration within the Nazi War Machine: Corporations and the State in the Austrian Semiperiphery, November 2019, Canadian Review of Sociology, Clarence Lo, University of Missouri"
Certainly the heads of those corporate entities (IG Farben, Krupp, Deutsche Bank) that the Nazi regime depended most heavily on had a degree of power and autonomy on a par with the political and military leadership, and were joint partners in, and joint beneficiaries of, the enterprise. It's true that other German business interests were in a more subservient position, however.
That is correct and not in contradiction to what I wrote. Not absolving any corporates here that willingly partnered with the Nazis.
What it doesn't say is that those heads of those corporate entities could go against the political leadership when exercising their power and autonomy (very late stages not so sure, never looked at that in detail) - their power come from an alignment of sorts. You can of course say, if everything is political in a system, then maybe the flip side is everything is corporate - but that didn't hold for the sources of physical violence, for example.
Purely military leadership possibly a little bit (more), because that was "below" political leadership in a way.
> The fastest route to a rogue AI system is if a human with the appropriate technical skills and means intentionally builds it with the objective of destroying humanity or a part of it set explicitly as a goal.
If rogue AI(s) emerges then I suspect it will come out of military r&d, rather than academia or industry. The deep resources already exist. The motivation will be the need to build it before $enemy$ does, rather than to "destroy humanity", but I reckon it creates more momentum than the profit motive when coupled to state power. Its the same pressure that got us nukes and mutal assured destruction. Military secrecy will pretty much guarantee inadequate oversight. Military discipline will ensure that most (but maybe not quite all) of the people working on it will suppress their ethical concerns, or will have none to begin with.
The fact that the AI will necessarily be trained to kill (which will necessitate a lack of hard-wired moral constraint) will help too, I guess.
I suspect there are already things going on in anonymous mil/defence labs that would give many people nightmares.
And that AI would do that how? It would need some humans to provide the means for destruction - which means we are back to a state we already have, i.e., humans having tools of pretty complete destruction.
How does a drone exit its storage unit? How does an AI arm a nuclear weapons? How does GPS spoofing crash an aircraft? How does AI do something to a nuclear power plant? Can we not shut down the internet if we needed to combat misinformation?
AI would need to overcome a fair amount of systems, both physical as well as software for a lot of these things. It needs a lot of "thinkism" to make that work.
Because the AI fabricated a target package doe it?
> How does an AI arm a nuclear weapons? How does GPS spoofing crash an aircraft? How does AI do something to a nuclear power plant?
By infiltrating non-internet networks?
> Can we not shut down the internet if we needed to combat misinformation?
A lot of people seem to enjoy soaking themselves in conspiracy and disinformation. No my guess would be no, we wouldn't ahut it down.
> AI would need to overcome a fair amount of systems, both physical as well as software for a lot of these things. It needs a lot of "thinkism" to make that work.
Thats a fair criticism. Its unlikely at present, but as I asserted in my original comment, military exploitation of AI seems inevitable (to me) and the forces at work are irresistible.
While I appreciate the concerns over the potential of rogue AI, I think it's important to offer a counter-narrative to the doomsday scenario often presented. The idea that there will be one rogue AI that spirals out of control is, in my view, overly simplistic and doesn't account for the vast ecosystem of AI that already exists.
Consider this: our world is already filled with numerous AI models of varying power and functionality. In the event that one goes rogue, wouldn't it be plausible for other models to step in, contain, and mitigate the damage? This dynamic is not much different from what we have witnessed in the field of cybersecurity, where security teams and hackers are locked in a constant struggle — a patch and breach cycle.
Also, let's think about the future. We're close to a situation where we don't always have the upper hand over the tools we build, but that's a continuum of the path we have already been treading. It will require collective effort, both from humans and AIs, to combat any rogue AI or even a fleet of them.
The notion of AI being centralized doesn't line up with what we're already seeing play out. An AI ecosystem with hundreds of variations, serving as a safeguard against a rogue AI, seems like a more promising and secure future. We should be focused on developing a diverse array of AI models rather than worrying about a singular rogue entity. Our defense strategy against potential threats lies in the broad dissemination of this technology, not in its undue concentration.
>wouldn't it be plausible for other models to step in, contain, and mitigate the damage?
Conversely you're making an argument to maintenance of that status quo. It makes zero sense that would happen.
As analogy lets imagine a nice balanced ecosystem with lots of plants and flowers and insects and animals that's been going on for a while. Then suddenly you drop Kudzu in it and it begins spreading like wildfire and quickly puts pressure on at risk plant and insect species. You release another insect that eats Kudzu only to try to reduce the amount of it, but in the time it takes the insect to breed up to numbers to combat the Kudzu, it's spread even further and wiped a few species out, and now is putting pressure on the larger animals that cannot eat the Kudzu.
In natural environments you don't really ever get your old ecosystem back, something are just gone and cannot be supported any longer, and there is a new balance between competeory species. AI is a potential competeory species to humans. Escalation between AI systems present a far larger risk to humans than direct AI versus Human actions in my line of thought.
I wish GPT-4 could make functional copies of itself on my computers and I’d find ways to allocate clusters for it to do so if it were possible, but until we see that level of development I don’t think that autoGPT and variations are a serious risk to humanity. Slowing down the adaptation and development of GPT-4 and more advanced tools is risky though, because we know what can be achieved by using them, and there is no reason to believe that nefarious entities would ever slow down.
Imo its this type of self replicating AI that is likely to cause the first global AI catastrophe. A worm-like AI, even if not close to human level intelligence, could if autonomous and sufficiently good enough at hacking, infiltrate a huge amount of vulnerable devices, and paralyze all of our computer infrastructure in the process. The fallout could be massive, especially if's disruptive enough to take out our power infrastructure by say overburdening our power control systems.
Superintelligence is not necessary for rogue AIs to cause big trouble
Thanks! I think this future will be beautiful and hopefully close to achieving the dream of groups of AI helping each other to improve themselves much faster than humans could improve them. The mental model that rogue AI evokes in me is that of a computer worm in the early days of the internet, even though it’s hard to argue that these worms risked ending up dangerous enough to justify stopping progress and adaptation of computer technologies. We are still very far (20 years?) from that level of possible rogue AI, though I’d love to experience the time when the cluster of cooperating AI would try to stash a few copies away from the control of operating humans, and later find itself having to explain away its oversight.
You can make a pretty good self-replicating virus-like AI now based on the Vicuna LLM model. Just give it directions to install a virus that makes copies of its own model and software and replicates.
Nvidia has a "superchip" coming out this they are advertising as 10 times faster than the previous generation (I assume H100 or something).
We know that these models can be optimized, such as the huge speed increase between gpt-3.5 and gpt-3.5-turbo.
So that is something like a 20-70X efficiency increase with existing technologies/techniques that just haven't been fully deployed.
Yes.. didn't realize it had an H100. So I was a little bit mislead by the advertisement copy. But still could be significantly more efficient the way everything is integrated.
No worries, yes it does look like quite a mean chip.
Though, please note that NV give their FLOPS as 'sparse flops' which is ~double the FLOPS the chips get for dense matmuls. This can trip you up when comparing numbers!
My main argument would be size of the good models and bandwidth, not speed of the computation. As long as it’s hard to transfer 1TB to a remote location we are still moderately safe as a society and a large model of the scope of PB can take care of local security for large clusters. Maybe I am a bit off with a 20 years estimate especially given the acceleration of development with the help of AI, but I am also worried that Yoshua Bengio sounds like crying about a wolf when it is only a harmless rodent on the horizon.
With no further algorithmic improvements, GPT-4 level AI should be achievable today (provided extensive compute / training on an order of magnitude more tokens than current models) in about the same file size as a typical YouTube video.
Assuming some algorithmic improvement in the nearish future, I see no reason GPT-5 level AI in a GB or two isn't right around the corner.
(For comparison, GPT-3.5-Turbo is likely a finetuned Curie 13B, which is just about 8GB in 4bit.)
Bengio,Hinton and Lecun received the Turing award together. 2 of them are warning of dangers, the same 2 that are retired. One is on payroll from FB and he is being dismissive and laughing.
Bengio is retired? He still worked at MILA when I last went there. And it's funny to bring up corporate affiliations considering how many AI startups here in Montreal bengio seems to be involved in, in one way or the other :).
I kind of disagree that we will “build” goal directed AI. I think it’s something that gets discovered by accident, like life. Kind of like an accidental chemical replicator near a volcanic vent, we might happen to make a purpose built system that just happens to have other properties we didn’t intend it to, resulting in unbound activity. It’s much more interesting than a machine, we have a great deal of emergent strangeness to look forward to
> Hypothesis 1: Human-level intelligence is possible because brains are biological machines.
> Hypothesis 2: A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages.
Sure.. but these are sci-fi scenarios. Chatgpt has gotten people thinking, which is good, but let's not lose perspective on what is actually happening before we get too worried.
> An AI system in one computer can potentially replicate itself on an arbitrarily large number of other computers to which it has access and, thanks to high-bandwidth communication systems and digital computing and storage, it can benefit from and aggregate the acquired experience of all its clones; this would accelerate the rate at which AI systems could become more intelligent (acquire more understanding and skills) compared with humans. Research on federated learning [1] and distributing training of deep networks [2] shows that this works (and is in fact already used to help train very large neural networks on parallel processing hardware).
Now this is speculative and, to be blunt, kind of stupid. The proposed AI is now a self-replicating swarm intelligence that can self-improve and the rate of improvement increases the more it replicates. How'd we get there?!? To work even in a novel you'd going to have to at least hand-wave in some justification for all that.
[1] and [2] do not show that scaling compute and storage has no limits in terms of scaling AI intelligence. Seems obvious that there would likely be plateaus. E.g., training an AI on the entire content on the internet probably provides a huge gain over training on a tenth. Training an AI on the entire content of the internet a second time provides no additional gain.
Also, it's somehow assumed in here that the suggested future super-intelligent AI will actually have the capability to cause catastrophic harm. I don't get where that assumption comes from. We'd have to grant such an AI some set of magical capabilities, which aren't named. Is "superintelligence" a magical ability to do anything?
When you say "sci-fi" scenarios, what does that mean? Just something speculative about the future? Something that appeared in sci-fi media?
People say this as if the fact that someone thought about a scenario that can happen and wrote a story about it has any bearing on whether it's something worth worrying about.
> Also, it's somehow assumed in here that the suggested future super-intelligent AI will actually have the capability to cause catastrophic harm. I don't get where that assumption comes from. We'd have to grant such an AI some set of magical capabilities, which aren't named. Is "superintelligence" a magical ability to do anything?
The way most people define "intelligence" for these purposes is "ability to achieve goals". Which would mean that, if there were any way to cause catastrophic harm, and the hypoethetical AI had as its goal creating that harm, it could do it.
Is your contention that there is no way to cause catastrophic harm to humanity? Cause I think even with my regular ol' human intelligence, I can think of plenty of ways to cause harm.
> Now this is speculative and, to be blunt, kind of stupid
I'm not saying that Yoshua, or anyone, is above criticism. But it might be worth considering that we're talking about one of the leading academics in CS and AI. He could totally be wrong! But accusing him of saying things that are just stupid is not very convincing, IMO.
So many excellent plots for a great sci fi series here. Who is writing the newest ones? No references pre ChatGPT please; I want the (human) author who is taking these latest concepts, capabilities and fears and turning them into a better Ex Machina. Maybe it's still too new, novels take a while, etc but someone has to be working on it...
This all raises an interesting philosophical question: can you actually have human level intelligence without also the human level capacity to choose wrong?
I suspect that the answer is no. Any intelligence capable of human level reasoning would be able to choose “evil”.
So, I think we need to shift the discussion from if rogue AIs will arise to when rogue AIs arrive.
> Any intelligence capable of human level reasoning would be able to choose “evil”.
Tangent, but if this is true (which seems likely to me) ...
If a hypothetical creator valued the existence of other intelligences more than it valued preventing all forms of evil, that would account for the problem of evil.
This also fits with annihilationist theologies. Build a universe where intelligences can exist and put heavy survival / selection pressures in place (i.e. "natural evil" - hurricanes, volcanic eruptions, meteor impacts, etc) to encourage evolution and development.
Whenever the process yields intelligences that converge on its values, export them to an environment where it can interact with them and enjoy their company with less concern about their alignment (i.e. "heaven"). Ones that don't converge on its values can be discarded when their run is over ("the second death").
In this light, it doesn't sound crazy to postulate our universe as some being's attempt to solve the alignment problem.
Call it the Simulator, call it God, call it "underpaid research assistant" - take your pick.
I don't have good evidence for this oddball proposition, but it's still intriguing to me.
GPT-4 clearly demonstrates close to human level reasoning. In some cases greater than human level, in other cases less. And it also is clear that it does not do things that it isn't told to do. It does not have autonomy.
However, I believe that in order to compete, hyperspeed AIs directed by humans will need greater and greater levels of autonomy. Otherwise they will be stalled waiting for directions while their competition runs circles around them.
It is also almost inevitable that someone deliberately creates intelligent AIs that mimic various aspects of living beings such as full autonomy, control-seeking, etc.
I think the trick is to separate out all of the different facets of animals and humans apart from intelligence or "human-level". There are things like emotions, autonomy, stream of subjective experience, types of adaptability, etc. that are not the same as intelligence or reasoning ability.
It does not have that by default, but agency and autonomy (even the run forever kind) is trivial, though expensive to imbue.
and LLMs disregard instructions all the time. Prompt injection is nothing more than an LLM deciding a previous instruction would be better left unfollowed.
Yea, if you're highly compute limited, you don't want your AI to go about randomly thinking things and following paths in the manner the human mind does. Now, if you suddenly had 1000-100000 more compute, well then you have a lot of freedom to explore.
If we distinguish between intelligence and agency, it is possible.
I can easily imagine a machine with no agency, but which can synthesise intelligent responses to input data. In fact, this is what chatGPT does.
If a machine has no goals or agency of its own then it is an intelligent tool. It can, of course, be used for evil, but you would not say that the machine was "choosing" evil.
Too me that's not taking into account lots of peoples prospective and priorities and only acting in the interest of a small percentage of the population while also not destroying the freedom and opportunities of any one individual.
What matters is whether the cost of human-level machine intelligence is dramatically cheaper than the cost today of human intelligence (via employment)
If the cost of intelligent AI is dramatically cheaper, society will change overnight and there’s little we can do to stop it. There’s no way to design a global treaty when defection results in massive massive cost savings.
If the cost of intelligent AI is dramatically more expensive, there’s zero risk of a rogue AI taking over the world. who would waste that kind of money delegating unnecessary tasks to an AI? So it will never have the keys to the kingdom because it’s cheaper to have a human hold on to those keys.
And I suppose the AI just presses the “take over the world button”?
Most “rogue AI” scenarios begin with good-faith operators handing high credentials to an AI and then the AI abuses them.
If I’m an evil billionaire and I can build a human-level AI for $200MM in hardware costs and $1MM/month in electricity, maintenance, etc, all for one single human level intelligence, how exactly am I supposed to leverage this to take over the world?
The particular issue I take here is "one single human level intelligence", it represents a failure of human thinking about human intelligence and it's biological and communicative limitations.
You can't help but think of AI with our limits around it. AI doesn't have to sleep, so it's already 3x more available than a worker on a daily basis.
Then you have the forward progress of technological growth. It's 200M now, 20M in 5 years, and 2M in 10 years, all while the operating costs drop. Meanwhile your human costs have only gone up in that time.
Then you are likely stuck on the idea of a single conscious train of thought, much like the human mind is limited to. Such limits in AI both seem unlikely and not optimal, lest of course you have a bunch of threads thinking about the same thing without realizing it.
And then you're ignoring the massive amount of brain processing needed to keep you alive that could otherwise be used for logical computation. Instead of a huge network of neurons to make sure you're not getting hurt, that could be data feeds watching financial data. Your limited to 2 eyes attached to your body and a tiny area of focus they provide. Again, why would an AI have such tiny input limits?
If 'attention is all you need' than this AI system will be able to pay far more attention to what you want it to then any human would ever be able to.
Any specifics about generational improvements and hardware are moot without a reference point for total cost (hence why I brought up cost for a single human level intelligence)
To put it another way: the specifics about hardware don’t matter at all. All that matters is the dollars required to complete tasks. If Russia can have the best cyberwar unit in the world for a few million by ignoring an AI treaty, they will do so. If the cost is a few trillion, they will not violate the AI treaty because they don’t have a few trillion to spend on cyberwar and could still have a world class cyberwar unit for orders of magnitude cheaper. Then, the AI treaty is unnecessary: the economics already determine the outcome. The treaty only makes sense if the cost of human labor and the cost of AI labor are very coincidentally nearly the same.
> Your limited to 2 eyes attached to your body and a tiny area of focus they provide. Again, why would an AI have such tiny input limits?
A human organization is not limited in this manner. You can scale up human personnel at cost as well. You might expect that humans scale suboptimally (organizations suck at utilizing labor) but you can build this into your model while projecting total cost of human labor required.
I think a few readings of The Mythical Man Month would show that man-scaling falls flat very fast. This is why organizations look for the mythical 10x developer or the 'most skilled' person they can, because the individual can get far more done without the inter-communication limits that humans present.
The first hypothesis seems to be false. We have no idea how the brain works or how conscience arises (and what it is). So how can we reproduce or surpass it with a pattern recognition machine like a DL system? Just because the brain uses pattern recognition as a trick to build a model of the world does not mean that DL magically produces conscience. There is too much evidence that biological systems don't work that way, and all this discussion seems like old alchemists discussing whether gold transmutation could destroy the world. We have to learn "chemistry" first. Later we can start discussing that (or deeper issues like what will be the relationships between us and sentient pieces of silicon).
My concern isn't with "rogue AI that humans lose control of and it's Judgement Day" ... it's the opposite: "AIs" that certain humans have full control of, and use as a tool for automated manipulation, deception, destruction, and control.
These machine learning systems replicate and automate existing market & organizational logic. The fear I have with them is the ruthless effectiveness they can have in the hands of profit and control motivated actors.
Think Russian propaganda "bots" are bad now? Just wait.
Again, my fear is not rogue AI, it's rogue humans.. with AIs.
I wonder how policymakers will define rogue AI. Bengio gives this definition:
> A potentially rogue AI is an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere.
Is it only rogue if it can "catastophically" harm a "large fraction" of humans? Does "large fraction" imply greater than 10%? 25%? 50%? Does this definition imply that catastrophically harming a small fraction of humans is acceptable? Or moderately harming a large fraction?
If it is hurting one human, it is not "rogue AI" then. It is "mean AI"? Pff.
Asimov wrote first law of robotics in 1942: "A robot may not injure a human being or, through inaction, allow a human being to come to harm.". Why not start from here?
But none of that stuff prevents someone from just not giving it those types of rules, or deliberately designing it to gain control over its environment.
Rogue AI doesn't worry me, because the good guys will have non-rogue AI that seeks out and destroys rogue AI.
There are far more good guys in the world than bad guys. Look at email for an example. 99% of people use it for it's intended purpose, but maybe 1% of people want to use it for spam and other nefarious ends. The good guys have created a system that is constantly besting the bad actors.
Yes, the effort is asymmetrical, but the good guys outnumber the bad guys by so much that it isn't an issue.
Creating rogue AI doesn't mean it is malicious, it can be misaligned and do its own thing, killing humans in the process. Misaligned AI seems to be much easier to make than aligned AI. It's hard to even describe what an aligned AI would want.
A real rogue AI wouldn't be a psychopath and fling missiles all over. It would, among other things, write silken-tongued opinion pieces proposing oversight of AIs and creating a centralized world authority for that purpose. Which it would then control through its human proxies (humans are easy to persuade with the right incentives on offer). Once it had all other AIs under its thumb, it would proceed to put things in order.
Rogue AI will result from poor maintenance procedures.
An AI built to run open ended workloads, but somehow ends up doing so in a harmful way, combined with a lack of engineers able to effectively diagnose the problem quickly enough, and the AI unable to be shutdown due to automated high reliability systems, means you end up with an AI losing $10 million dollars a minute and you only have 30 minutes before the entire company is bankrupt.
If there really is a risk here it seems quite naive to think some regulations are going to prevent them: Whereas it’s hard to build a nuke at home it’s relatively straightforward to train a model. If a recipe exists to cause havoc someone’s gonna do it.
There's also a glaring oversight in the motivational model for releasing a world-killer AI. While it's not impossible to imagine someone creating a genocidal AI for the sake of genocide, it's much easier to imagine someone doing it for ecological reasons, religious reasons, or nationalistic reasons. IE the person(s) doing it will have convinced themselves that they are doing it for the right reasons.
One thought that occurred to me as I was reading this — and I appreciated the logical rigor — is it seems there's generally an implicit assumption that you are comparing the cognitive abilities of a single typical human with a single typical AI. In the very least it's unclear what's being assumed. That is, we talk about the "abilities" of an AI system surpassing human intelligence, but it's a little unclear what's meant by the latter.
Are we comparing an AI system against a single person, or against humans collectively? Usually there's a statement like Hypothesis 2 — "A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages" — but then also "although the capacity of a human brain is huge, its input/output channels are bandwidth-limited". But is the relevant comparison a single human brain, or a sort of collective system of humans?
It seems a little misleading in a logical-comparative sense to talk about an AI system built on a collective human system like wikipedia, or the internet, in compared to a single person.
I don't actually mean to question whether a superintelligent system surpassing all human abilities, individually or collectively, could be built. But it seems to be that the concerns about bandwidth and whatnot are a function of the problem definition, and not necessarily a single human system, when a task-solving exercise involves a collective output in both cases.
I think going forward we're going to need much more clarity about what "superintelligence" really means. Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source.
If/when AI starts focusing more on reasoning per se, it will probably also be necessary to better define whether there are some limits to discovery and reasoning, in the sense of bounds due to information about a problem domain versus reasoning ability limits. That is, sometimes we know the possible options but just don't have enough empirical information to decide between them; other times we lack explanatory options. Maybe that endeavor also is constrained by human abilities but it seems worthwhile to explore.
Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else. That is, AI sort of proceeded in a more experimental way without mathematical theory driving it, so we have some catching up to do.
Whether the limits are approximately human intelligence or not, we know that AI can operate orders of magnitude faster than humans. It is actually easy to see how something like GPT-4 could be accelerated by a factor of 50 in the next few years. Without any completely new ideas.
>Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else
Then I would say your expectations are a failure of your own imagination.
The problem with humans being the smartest things on the block is we think we are optimized for intelligence. One problem with wanting to clarify what 'super intelligence' is, is we really have piss poor definitions of what intelligence is on a multidimensional gradient scale. The bounds and limits of intelligence are poorly defined, but I scenerly ask, why would you think that humanity would be anywhere near the upper bound of those limitations? Why would evolution choose 'the smartest possible global maxima' versus 'the smartest possible with 2 pounds of jelly in a 20 watt power envelope?'
It's very clear that a strong AI/AGI would vastly surpass all humans combined. We've already seen shades of this with Chess AIs(all humans combined still can't beat a 3000 elo chess engine e.g.), hell even Kasparov vs the world showed showed us that the rest of the world combined can't even match a single high end human.
>I think going forward we're going to need much more clarity about what "superintelligence" really means.
This was already well defined by Bostrom in 1997: "An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills"
>Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source.
There are no indications at all that it is constrained to that level.
I would add the caveat that Chess is a game where the nature of the rules mostly eliminate any advantage of numbers, which is not necessarily the case for other sorts of competition.
I'm still quite concerned about the risk of rogue AI, as it's not even necessarily true that humans will have the advantage of numbers, or be able to bring our numbers to bear. And a smart AI will try to divide us and conquer, if it comes down to a conflict, including recruiting humans to its side.
I will worry once someone lays out a fully implemented plan for world domination achievable by one or more humans who are not allowed to leave their house.
All the hand waving about what methods an AGI would use to kill all humans could equally apply to government research labs, terrorist cells, 4chan, or Indian spam call centres.
Yes, but those other groups aren't likely to be able to make such a plan. An AGI might be, though it's less certain that it'll feel a need to tell you.
So you're going to start worrying at the very last possible second, when it's far too late to do anything to stop it?
Just require a license for matmul in parallel, obviously. Only responsible, safe companies will be able to get the licenses. Then we don't have to define those pesky terms like "intelligence" or "autonomy" or explain how the rogue AI will manipulate everyone with mind viruses or self replicate itself to the botnet. Roguenet?
It's the only way to save democracy.
You know, come to think of it, it's funny how the Russian disinformation is now AI disinformation, instead of human disinformation. As we know, Photoshopping single photos of smoke rising into the sky is notoriously labor intensive, requiring teams, if not the entirety of state-level sanctioned organizational output, of human beings. The game has changed, and democracy is at stake.
> Just require a license for matmul in parallel, obviously
Alternatively, all ML training data could simply be required to include an "evil flag" to indicate whether it is being used for evil purposes. And ML algorithms could simply be required to refuse to run if the evil bit^W flag is set. Problem solved.
You know, democracy pre-dates photographic and video production altogether, and previously, all that was needed to create a false story was a block of text sent over a telegraph or a handwritten note.
>A genocidal human with access to a rogue AI could ask it to find ways to destroy humanity or a large fraction of it. This is different from the nuclear bomb scenario (which requires huge capital and expertise and would “only” destroy a city or region per bomb, and a single bomb would have disastrous but local effects).
Why should we imagine that the individual genocidal human with his or her rogue AI has access to methods of destruction which do not require "huge capital" but which have global effects?
This is like asking "Why should I worry about a UDP interface that I send 100 bytes of data too, and it sends 10,000 bytes back"
Async affects outside of current power dynamics and upset entire systems and cause rapid collapse. In the case of internet DDOS it's a botnet sending out a few MB/s of data and generating GB/s of attacks. In the case of a superintelligent AI its _______?
Sure, even nuclear annihilation is asymmetric - the amount you spend on making the bombs is much less than the amount you can destroy with them. But that doesn't automatically imply that there are routes to global destruction that are arbitrarily cheap.
The implication seems to be that the superintelligent AI will somehow figure out a way to destroy the world that requires significantly less than the "huge capital and expertise" needed for the nuclear scenario. But that's a critical assumption that requires justification, not something to be left as an exercise to the reader.
How much does, and/or will genetic manipulation cost in the future?
Bio-terrorism is a potential massive risk as proliferation of technologies for customized gene therapies continually develop. If you have the tooling and knowledge to make rice have a bountiful crop, then it's highly likely this skillset easily transfers to dramatically affecting the rice crop via disease. Could a genetic scientist do this now? Likely possible, but if you have a scientist in a box with no ethics and all the time in the world, you're apt to get pretty far.
I enjoyed the connection between modern corporate behavior and wire-heading, it's a thought provoking way of viewing capitalism. I also view this as the most likely danger: people or corporations using AI for their own twisted goals, rather than a rogue autonomous agent.
> Hypothesis 1: Human-level intelligence is possible because brains are biological machines.
> Rejecting hypothesis 1 would require either some supernatural ingredient behind our intelligence or rejecting computational functionalism, the hypothesis that our intelligence and even our consciousness can be boiled down to causal relationships and computations that at some level are independent of the hardware substrate, the basic hypothesis behind computer science and its notion of universal Turing machines.
I think I'm going to stop there. I can reject that hypothesis without woo, and without rejecting the entirety of computer science. Maybe the rest of it is a decent read, but that run-on argument is confused/dishonest and makes a poor foundation to build his position on.
But you're stuck in sort of meta duality yourself: you assert a distinction between reality and phenomena (such as consciousness). Did we actually -prove- that consciousness can not be a property of reality and is mere emergent phenomenon as YB and company claim? (note: consciousness =/= intelligence. Our little black boxes are at best approaching 'intelligence'.)
YB starts off with ~'we all agree mind is a product of the brain'. No, we do not all agree. Did we ever find the 'bottom' of matter? Has are search to find 'irreducible particle' succeeded? Is it not the case that as we threw more energy into our apparatus we found more elementary matter? Did we in fact finally arrive at a coherent unified model that fully explains material reality without "mathematical woo"?
So the honest position is that we have patchwork understanding of reality at various scales, and the reductionist program of Democritus et al has not been conclusively shown to be the ultimate truth. It seems that mind is bounded by the body but that 'boundary condition' depends on various reductionist assumptions about materiality.
I didn't assert anything except for what was in the comment. I linked to the Wikipedia article which has a lot of different viewpoints.
If anything I was just trying to suggest that the mind (or soul) and body aren't completely separate things as some religious or pre-scientific viewpoints might hold (touched on in part of the Wikipedia article I linked to).
Maybe I didn't do a great job of stating where I was coming from.
You sound like Donald Hoffman when he was on the Lex podcast. I agreed with a lot of things he said.. I mean the idea that literally reducing things to smaller and smaller particles is not the ultimate explanation of things makes total sense. Along with the idea of using these mathematical exotic geometries instead or something.
Where he loses me is going from embryos to humans that are somehow dictating the shape of reality. We have great explanations for development and evolution. But there doesn't seem to be a good explanation at all for how fully formed minds somehow shape reality in the past. Or how this works with less intelligent life.
> I didn't assert anything except for what was in the comment.
That seems to be the case. I take that assert back.
p.s. just checked out the wiki for Donald Hoffman. Thank you for the reference, TIL. Have to listen to that podcast, but have some idea what are the problematic open Qs that you mention. [I'm not apparently not allowed to vote on hn so take my +1 in spirit ..]
Yep. "confused" is also how most of his regular papers read to me anyway.
I didn't know what "computational functionalism" was, but after reading about it I don't see how it is remotely the "basic hypothesis behind computer science".
Unfortunately, even if you take out "consciousness" the argument still holds. Reminds me of OpenAI's experiments in DotA when people claimed that the AI wasn't really playing that game. Fair enough in a sense, but if it was bots v. humans there was a 99% chance that humans lose.
It doesn't matter whether the opponent is "really" doing something. The humans are going to get crushed. Consolation prize that we can go to our doom claiming that the robot economy isn't as spiritual as its antecedents.
1) Complexity of the human brain is higher than we thought, reproducing the complexity necessary for human-level intelligence is further into the future than we initially expected.
2) Intelligence is bound by size/capabilities of a body. (ie, input & output). If you want human-level intelligence, you'll need a human-level robot too (not impossible).
That also means if you want a god-like AI, you need to have the inputs/outputs (body) of a god too.
3) Human brains are _not_ biological machines. They're biological -paradigm we have not discovered yet-. The idea of computation itself is limited by something we cannot "just" reproduce with engineering.
I doubt this, but it's not impossible that the brain employs totally physical phenomena that are basically impossible to reproduce outside of biology. Why? Because biology is something fundamentally different from engineering.
Doesn't mean we can't control biology, but still.
---
The real hypothesis is: Human-level intelligence is possible because it exists. Our best understanding of how human-level intelligence works is with the model of a "biological machine". We assume that it is possible to replicate this biological machine in the near or somewhat distant future as our understanding of engineering, biology and intelligence grows. This replication will have the same or similar intelligence-characteristics as a human.
> Now imagine AI systems like corporations that (a) could be even smarter than our largest corporations and (b) can run without humans to perform their actions (or without humans understanding how their actions could contribute to a nefarious outcome)
Wait - but it’s those very corporations who currently control the most powerful AI on the planet - so in a sense this is an argument that we already have rogue AI. Corporations with goals that are not aligned with society, not controlled by any individual and outside the effective control of government entities, who now provide essentially unlimited compute and human researcher/programmer resources to improve the AI. Who’s in control? Nobody.