Most functional AI nowadays consists of algorithms that are carefully tuned to solve a very specific problem in a narrowly defined environment. All the research nowadays is pushing the boundaries of a local optimum. Right now, true AI is a pipe dream without a fundamental shift in how we approach AI-type problems. And no, machine learning/deep learning is not that shift; it is just another flavor of the same statistics that everybody already uses.

What concerns me is not Skynet; what concerns me is the exasperating over-confidence that some people have in our current AI capabilities, on Hacker News and elsewhere. Too often, we discuss such technology as a miracle tonic to various economic or social woes, but without acknowledging the current state of the technology and its limitations (or being completely ignorant of such), we might as well be discussing Star Trek transporters. And usually, the discussion veers into Star Trek territory. Proponents of self-driving cars: I AM LOOKING AT YOU.

Take self-driving cars: at least with humans, our failure modes are well-known. We cannot say the same for most software, especially software that relies on a fundamentally heuristic layer as input to the control system. To that mix, add extremely dynamic and completely unpredictable driving conditions -- tread lightly.

Again, that's skirting the issue. Do you have any idea how close self-driving cars are to being "better than us" ? As someone who's done computer vision research: not close at all.

> I don't think the computers will ever match our judgement

That is exactly the problem.

> it's trivially easy for them to beat us on attention span and reaction time

Attention span and reaction time are not the hard parts of building an autonomous vehicle.

This kind of comment beautifully illustrates the problem with casual discussions about AI technology. Humans and computers have very different operating characteristics, and discussions all focus on the wrong things: typically, they look at human weaknesses, and emphasize where computers are obviously, trivially superior. What about the contrapositive: the gap between where computers are weak, and where humans are vastly superior? More importantly, what is the actual state of that gap? That question is often completely ignored, or dismissed outright. Which is disappointing, especially among a technically literate audience such as HN.

Don't forget some people speed, flee from the cops, fall asleep at the wheel, get drunk, text, look at maps, have strokes etc. So, sure at peak performance cars have a long way to go. However, accidents are often a worst case and computers are very good at paying attention to boring things for long periods of time.

PS: If driverless cars on average killed 1 person each year per 20,000 cars then they would be significantly safer than human drivers.

Again you are falling into the same pit: a nonsensical comparison of human and computer operational/failure modes. Of course computers can't have strokes. And yes, they are good at "paying attention to boring things". That is trivially true. And that's not where the discussion should be focused.

I do hope self-driving cars will be generally available sooner rather than later. What's not to like about them? But what I'm really curious about is how that availability will be qualified. Weather, road, visibility conditions? Heavy construction? Detours? Will this work in rural areas or countries that don't have consistent markings (or even paved roads!)? Will a driver still have to be at the wheel, and what extent will the driver have to be involved?

What is really annoying are breathless pronouncements about a technology without critically thinking about its actual state and implementation. We might as well be talking about Star Trek transporters.

Basicly an MVP that flat out refuses to operate on non designated routes, bad weather, or even highway speeds could still be very useful.

PS: Classic stop and go traffic is another area where speeds are low, conditions are generally good. But, because people can't pay attention to boring things you regularly see traffic accidents which creat massive gridlock cost people day's per year sitting in traffic.

Based on what? Even Google admits that the car is essentially blind (~30 feet visibility) in light rain. They've done little to no road testing in poor weather conditions. The vast majority of their "total miles driven" are highway miles in good weather, with the tricky city-driving bits at either end taken over by humans.

Google often leaves the impression that, as a Google executive once wrote, the cars can “drive anywhere a car can legally drive.” However, that’s true only if intricate preparations have been made beforehand, with the car’s exact route, including driveways, extensively mapped. Data from multiple passes by a special sensor vehicle must later be pored over, meter by meter, by both computers and humans. It’s vastly more effort than what’s needed for Google Maps.

http://www.technologyreview.com/news/530276/hidden-obstacles...

No, I'm not drinking the AI koolaid here thinking there will be some breakout solution to fuzzy visibility problems these cars have. The difficulty differentiating contextual moving objects correctly, knowing what is "safe" debris and what is not, etc, are all neural net problems that will take years of development and even then just be heuristics.

But you don't really need all that. All you need is something that, given a sunny day and a highway, beats the human at driving the truck, and suddenly you lose three million jobs when it hits the market.

I don't see how, given a sunny day and a highway, an autonomous vehicle can quantifiably beat a human driver to the degree of 'destroying the trucker industry overnight.'

Unless you're assuming that there are few human drivers capable of managing a trip down a highway under the best possible conditions, the results between the two would have to be more or less equal. Those human drivers can, meanwhile, still manage night, sleet, snow, fog, and arbitrary backroads and detours.

Destroying the trucker industry with a self-driving car which can't drive in bad weather?? I don't think so! Trucker's clients usually care a lot about predictability, and they would NOT be happy to hear "sorry it is raining and the robot car couldn't arrive".

The things that the car needs to know are largely standardized: the lines on the road, road signs, speed limits, etc.

These are things humans need to know as well, and yet autonomous car cheerleaders constantly argue that human drivers are death machines, despite the fact that most human drivers know perfectly well how to follow these norms, and even deviate from them, without incident, most of the time.

I suspect that in their enthusiasm to set the bar of human intelligence as low as possible in order to make the case for autonomous cars seem urgent and necessary, some vastly underestimate the actual complexity of the problem. An autonomous car that only knows to 'avoid the boxes' has worse AI than many modern video games.

What you've basically described is that mainstream "AI" is actually quite divorced from what the public thinks of as "AI" -- something matching or improving upon human-level intelligence. Still, there are those still working on original grand dream of artificial intelligence.

[1] http://agi-conference.org/ [2] http://www.agi-society.org/journal/

The problem with these names are that they are not easily distinguishable by the general public or others who are not already familiar with the concepts. Its like the poor coding pattern where you have two variables in your code with almost the same names - 'wfieldt' and 'wtfield' will get confused by the next person who has to maintain your code. It's the same problem with popularization of concepts.

The same thing happens in Physics, where people confuse energy with power, and think that Special Relativity must be much harder than General Relativity (it's special, right?). The words used to name things make them understandable, or not.

What's worse is that this confusion is intentional, created on the part of people who wanted to hype their (actually valuable, but perhaps overlooked) work, by confusing their 'boring' or 'limited' AI work with AGI work in the minds of those supporting or funding that work.

That's why people say things like "true AI".

There really needs to be a name for the non-AGI kind of AI, which refers specifically to the limited-domain type of problem solving which "AI" has been so successful. "AI" is a bad name for this.

Personally, I am partial to the metaphor of "canned thought". It shows that a human had to do the work beforehand of setting up the structure of the algorithm that is later doing the information processing, and that the process is limited by the thought that were previously put into it. But Canned Thought is also a pretty bad name for the field. It's not descriptive and it's not catchy. Anyone have any suggestions for a better name for non-AGI AI?

I think you are looking for Narrow Artificial Intelligence, which hasn't gained much traction. To me Narrow and General AI respectively make perfect sense for their scopes.

[1]http://en.wikipedia.org/wiki/Weak_AI

But sentience simply is not defined. There are no equations for it, no foundational science modeling it. It's possibly the ultimate unanswered question for all human history, and so the task given to computer engineers is: implement this thing we haven't as a species been able to explain or describe or comprehend for 10,000 years

And really, it's a bit much to ask. The neuroscientists could have a eureka moment tomorrow and finally crack the code of sentience, and once documented I'm confident some form of rudimentary synthetic implementation would be possible within a decade. But I'm not sure many neuroscientists or practically anyone (Tononi/Tegmark/Hawkins/?) are directly attempt to investigate and build theories of consciousness.

That theory of consciousness would be the instruction book for the programmers building the first hard AI.

Answer: 42.

I'm worried about the descendant of Bonzai Buddy. I'm worried about annoying little algorithms that are simple-minded but have access to vast quantities of data I'd rather they not, and that have means of actuating things in the real world that could prove annoying.

Everybody seems so goddamned thrilled over this stuff that nobody seems to be worried about the mundane but harmful things we can accomplish and enable with current-day technology.

Imagine a bot whose only job is to squat a hashtag and intermittently harass people who mention it.

Now stop imagining, because we've already got those.

Imagine another bot that just looks for bug trackers and spams incomprehensible combinations of issues generated by mining previous issues. It can be hard, on a good day, to tell difference between frustrated and inarticulate user, and markov-chain spam.

Let your imagination run wild.

Yes exactly!!

Each AI winter is preceded by unrealistic expectations. If the unrealistic expectations this time are resoundingly fearful, I worry that the next AI winter will be even worse - like create legislation and basically kill the field worse.

What you describe is one half of the Amara's Law: "We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run"

What you demonstrate with your comments is the other half :)

A lot of technologies were over-hyped first and underestimated later, until they just appeared to be part of our lives. I'm quite sure that, sooner or later, it'll happen with AI too.

If you assume these cars will some day be able to identify humans, what decision is made in a situation where the car can either swerve and (potentially) injure a bystander, or continue on a trajectory which calculates to be certain death for its occupants?

What shocks me is how few people understand that this would not be a "decision" in the sense that the software would have a choice. It would either be coded in directly or would be a resultant output of the criteria it uses to already do its pathfinding.

I don't think my own interest in AI recovered from reading Drew McDermott's A critique of pure reason - fortunately that was about 1992 and I found this really cool networked hypertext thing that was far more interesting....

http://karpathy.github.io/2012/10/22/state-of-computer-visio...

How do you think the human mind runs? Something other than statistics?

Theoretical neuroscience at least partially disagrees[1]

[1] -- http://www.fil.ion.ucl.ac.uk/~karl/The%20free-energy%20princ...

I worry that the flurry of articles like this, rational and well-reasoned all, will be seen as a "win" for the nothing-to-worry-about side of the argument and lead people to discount the entire issue. This article does a great job demonstrating the flaws in current AI techniques. It doesn't attempt to engage with the arguments of Stuart Russell, Nick Bostrom, Eliezer Yudkowsky, and others who are worried, not about current methods, but about what will happen when the time comes -- in ten, fifty, or a hundred years -- that AI does exceed general human intelligence. (refs: http://edge.org/conversation/the-myth-of-ai#26015, http://www.amazon.com/Superintelligence-Dangers-Strategies-N...)

This article rightly points out that advances like self-driving cars will have significant economic impact we'll need to deal with in the near future. That's not mutually exclusive with beginning to research ways to ensure that, as we start building more and more advanced systems, they are provably controllable and aligned with human values. These are two different problems to solve, on different timescales, both important and well worth the time and energy of smart people.

That's because it would be arguing a straw-man. So there is no reason to engage the argument.

By contrast, this article and many others are implicitly arguing against an actual straw man position, the position of "let's shut down AI research before it kills us all in six months", which no serious person on either side of this debate really holds (though it's understandable how someone who only read the discussion in the mainstream press could come to think this way).

There isn't really though. Superintelligence, while an interesting book, doesn't do much more than pontificate on sci-fi futures based largely on writing from Yudkowski. Bostrom gives no clear pathway to AGI. Neither does Yudkowski in his own writings or Barrat (Our final invention). By the way Super intelligence is kind of an offshoot book from Global Catastrophic Risks.

All of them take these interesting approaches, WBE, BCI, AGI etc... and just assume they will achieve the goal without looking realistically about where the field is or how it would get there.

So what they do is they say: We are here right now. In some possible fantasy world we could be there. The problem is they can't draw the dots between them.

For example, find me someone who can tell me what kind of hardware we need for an AGI. Can't do it, because no one has any idea. What about interface, what is the interface for an AGI?

Even better (this was my thesis project): What group would have a strong enough requirement that an AGI would be required to build? Industry, Academia, Military? Ok great, can they get funding or resources for it?

etc...

Note, I am not saying that there is no potential that AGI could be a problem. The point here is that nobody has firm footing to say that it will definitely be a huge risk and that we need to regulate it's development.

There are actually people calling for legislation to prevent/slow AGI development - see Altman's blog post and many of the writings on MIRI. So that is what I am rallying against.

We need more development into AGI not less.

Could you argue against the position directly? You kinda took a swerve in the middle there. Bostrom, Yudkowsky, and Barrat's positions are basically "superintelligence is possible, it will eventually happen in some way, and when it does ..." It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence.

So you would like to see unencumbered research in this area. They argue that the long term outcome of this research is existentially risky and therefore should be tightly regulated, and the sooner the better. Do you have any points regarding this particular difference of opinion?

No I can't because there isn't a way to argue it. I can't argue that it would be safe because I have nothing to point to that says it will be or even could be. We simply do not know enough about how it would be built to reason on it.

It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence.

Correct, but they don't even reference a technical road map because there isn't one. It's too early to tell.

Do you have any points regarding this particular difference of opinion?

Not really, other than my own opinions which are as baseless as the others. I mean I have my own opinions and thoughts on things but they aren't empirically based at all, which is my main contention. We can't reason, let alone start making policy on stuff that we haven't the slightest idea how to build.

But I don't think it's at all a fair criticism to say that you need a technical roadmap to engage in that debate. I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure. The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates.

We are in violent agreement then. When I say roadmap I certainly don't intend to mean that each step needs to be perfectly detailed.

The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates.

Exactly exactly and well said! That is 100% the point. Mind meld complete.

An alternate phrasing might state that it needs more philosophy and an even greater amount of engineering.

If the field is in its infancy from a technical standpoint, then it probably is from a philosophical one as well.

That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again.

Actually I am almost completely unpersuaded by the arguments of Bostrom, Yudkowsky, et al, at least in their presented strong form. Superintelligent AI is not a magical black box with infinite compute capacity, the two assumptions which really underlie the scary existential risk scenarios if you look closely. It is not at all clear to me that there will not exist mechanisms by which superintelligences can be contained or their proposed actions properly vetted. I have a number of ideas for this myself.

In the weak form it is true that for most of the history of AI its proponents have more or less ignored the issue of machine morality. There were people in the 90's and earlier who basically advocated for building superintelligent AI and then sitting back and letting it do its thing to enact a positive singularity. That indeed would have been crazy stupid. Kudos to Yudkowsky and others for pointing out that intelligence and morality (mostly?) orthogonal. But it's a huge unjustified leap from "machines don't necessarily share our values" to "the mere existence of a superintelligent amoral machine will inevitably result in the destruction of humanity." (not an actual quote)

Only the latter is used for programming important things like airplanes, missile silos, and other things that can kill people and cost millions or billions of dollars when they go wrong -- the things in which society knows we cannot afford software errors.

I don't think it's even remotely a leap to say that we ought to apply only this thorough school of engineering to artificial general intelligence, whether it's at the level of an idiotic teenager or whether it's a superintelligent scifi thingy.

Now, if we think we can't really solve the inherent philosophical questions involved in making a "world optimization" AI that cannot go wrong, then we should be thinking about ways to write the software so that it simply doesn't do any "world optimization", but instead just performs a set task on behalf of its human owners, and does nothing else at all.

But either way, I think leaving it up to New Jersey kludging is a Very Bad Idea.

At the very least, we have a clear research imperative to ensure that "AI", whatever we end up using that term to mean, "fails fuzzily" like a human being does: that a small mistake in programming or instructions only causes a small deviation from desired behavior.

The authors above are unpersuasive in arguing that these things are inevitable, but they are persuasive in arguing that these things are possible. And if they are possible, then what matters to me is their likelihood. Based on current estimates, are we talking about a 50-50 chance here? 80-20? 99-1? 99.999-0.001?

If we're talking about the future of the human race, these chances matter. But instead it seems everyone is just waving their hands at the what-ifs simply because they aren't 100%.

Security for human level threats is already very poor, and we already have a relatively good threat model. If you suppose that an AI could have radically different incentives and vectors than than a human, it seems implausible that you could be secure, even in practice. I suppose you could say that these would be implemented in time, but it's not at all clear to me that a humanity which has trouble coordinating to stop global warming or unilateral nuclear disarmament would recognize this in time.

On the other hand, I'm slightly puzzled by why you think there's a huge unjustified leap between lack of value alignment and threat to the human race. Does most of your objection lie in 1) the lack of threat any given superintelligent AI would pose, because they're not going to be that much smarter than humans or 2) the lack of worry that they'll do anything too harmful to humans, because they'll do something relatively harmless, like trade with us or go into outer space?

For 1, I buy that it'd be a lot smarter than humans, because even if it initially starts out as humanlike, it can copy itself and be productive for longer periods of time (imagine what you could do if you didn't have to sleep, or could eat more to avoid sleeping). And we know that any "superintelligent" machine can be at least as efficient as the smartest humans alive. I would still not want to be in a war against a nation full of von Neumanns on weapons development, Muhammads on foreign policy and Napoleons on Military strategy.

For 2... I would need to hear specifics on how their morality would be close enough to ours to be harmless. But judging by your posts cross thread this doesn't seem to be the main point.

By the way, I must thank you for your measured tone and willingness to engage on this issue. You seem to be familiar with some of the arguments, and perhaps just give different weights to them than I do. I've seen many gut level dismissals and I'm very happy to see that you're laying out your reasoning process.

So what's the point? I mean, you may want to form a group and discuss those scenarios, but it's not excusable to spread fear mongering in the media with no substance like that. It's unscientific in a way.

We don't really need to start writing traffic rules for flying cars.

1 - There's no real dispute that computers will eventually be as "powerful" as brains. In fact, it's likely that they will one day be far more powerful.

2 - Assuming the brain evolved naturally, there's no reason to assume humans won't eventually duplicate the software once we have the hardware. It's really only a matter of when.

3 - An AGI with equal/more power than the human brain will, eventually, be able change its own code and improve upon itself.

4 - Given the characteristics of intelligence as it has been observed thus far in nature, even small increases in IQ lead to massive improvements in capability that are difficult for lesser intelligences to comprehend. A sufficiently advanced AGI would not only be highly capable, but would quite possibly be too smart for us to predict how it might behave. This is a dangerous combination.

Further complicating things, we might hit step 2 on accident, without realizing that we hit it. Or some group might accomplish step 2 in secret.

What I'd like to see someone do is argue that the chance of these things happening so small and/or the consequences so minuscule that it's not worth worrying about or planning for.

General intelligence is, well, general. My primate brain may not be able at all to intuit what it is like to move on a 7-dimensional hyper-surface in an 11-dimensional space, but thanks to the wonders of formalized mathematics I can work out and tell you anything you want to know about higher-dimensional geometry. If the super-intelligence itself is computable, and we have the machinery to verify portions of the computation ourselves, it is in principle understandable.

Of course there will be computational limits, but that's hardly anything new. There is no single person who can tell you absolutely everything about how a Boeing 787 works. Not even the engineers that built the thing. They work in groups as a collective intelligence, and use cognitive artifacts (CAD systems, simulators, automated design) to enhance their productive capacity. But still, we manage to build and fly them safely and routinely.

There is no law of nature which says that human beings can't understand something which is smarter than them. Deep Blue is way smarter than me or any other human being at Chess. But its operation and move selection is no mystery to anyone who cares to investigate its inner workings.

A better analogy would be to stick to IQ. How long would it take a group of people with an average IQ of 70 to understand and predict the workings of the mind of a genius with an IQ of 210? Probably a very long time, if ever. What if the genius was involved in direct competition with the group of people? She'd be incentivized to obscure her intentions and run circles around the group, and she'd likely succeed at both.

Just how intelligent might an AGI become given enough compute power? A 3x IQ increase is a conservative prediction. 10x, 100x, or even 1000x aren't unimaginable. How can pretend to know what such an intelligence might think about or care about?

This is a great metaphor and one I wish I would have thought up sooner.

And, no surprise, we're seeing exactly this discussion going on right now about drones.

My guess: Something with tons of flops (or even fixed point ops, if that would be easier). Everything else secondary.

Look at the technologies underpinning the internet. We badly screwed up a lot of things that are very difficult to fix in retrospect, because we didn't invest enough time and effort into foreseeing the consequences.

Nobody worried about the fact that SMTP allows people to send millions of unsolicited messages at practically zero cost until it was far too late to fix the protocol. It is entirely plausible that an RFC describing a proof-of-work scheme could have fixed the problem from the outset, but now we're stuck with Bayesian filters and black/whitelists that sort of work acceptably most of the time.

The issues we're seeing regarding surveillance and censorship could have been hugely ameliorated if our protocols were designed with greater foresight, if people like Cerf and Berners-Lee were more aware of those risks.

We look back in horror at the naive techno-utopianism of the past and the harms that resulted - tetraethyl lead, asbestos, nuclear fission, CFCs. They were all heralded as miracles of the modern age, but caused harms that we're still dealing with today. The technology industry needs to be far more circumspect about the hazards of what we do. We need less hype about changing the world, and more sleepless nights about the people we might be harming.

We need to be having big discussions right now about every aspect of technology, before we open Pandora's box. For example, the World Anti-Doping Agency already have rules prohibiting genetic doping in sport, because they know that it's a potential risk. They would rather figure out their policies now in the cold light of day, rather than in a panic when the Russian team arrive at the 2032 Olympics with a team of superhuman mutants.

AGI is a starting point for broader discussions about how society deals with increasingly powerful and ubiquitous computing technologies. Maybe it will happen, maybe it won't, but I don't think that's particularly important; What we can all agree on is that machine intelligence will only become more disruptive to society.

We need to plan now for what we would do if certain technologies come good. Driverless cars might never happen or they might be five years away, but we need to figure out what to do with the millions of Americans who drive for a living before we put them all out of work. What would be the social and ethical implications of brain-computer interfaces, of algorithmic decision-making in medicine or the criminal justice system, of ubiquitous facial recognition and tracking? How can we plan to mitigate those harms? If we don't give serious thought to those kinds of questions, then we're sleepwalking into the future.

Wouldn't the human brain qualify?

To actually argue the point a little bit, the theory of expected-utility-maximizing agents is pretty much the framework in which all of mainstream AI research is situated. Yes, most current work is focused on tiny special cases, in limited domains, with a whole lot of tricks, hacks, and one-off implementations required to get decent results. You really do need a lot of specialized knowledge to be a successful researcher in deep learning, computer vision, probabilistic inference, robotics, etc. But almost all of that knowledge is ultimately in the service of trying to implement better and better approximations to optimal decision-theoretic agents. It's not an unreasonable question to ask, "what if this project succeeds?" (not at true optimality -- that's obviously excluded by computational hardness results -- but just at approximations that are as good or better than what the human brain does).

Do Nick Bostrom and Eliezer Yudkowsky understand when you would use a tanh vs rectified linear nonlinearity in a deep network? Do they know the relative merits of extended vs unscented Kalman filters, MCMC vs variational inference, gradient descent vs BFGS? I don't know, but I'd guess largely not. Is it relevant to their arguments? Not really. You can do a lot of interesting and clarifying reasoning about the behavior of agents at the decision-theoretic level of abstraction, without cluttering the argument with details of current techniques that may or may not be relevant to the limitations of whatever we eventually build.

The issue is still the same: if a system that doesn't perfectly share your goals is making decisions more effectively than you, it's cold comfort to tell yourself "this is just a control system, it's not really intelligent". As Gary Kasparov can confirm, a system with non-human reasoning patterns is still perfectly capable of beating you.

If we don't regard intelligence as something different from control, then I guess birds are the most intelligent because they can navigate complex air currents. Etc. That is a poor definition of intelligence, because its not helpful in distinguishing what we normally mean by 'smart' from mechanistic/logical systems.

And the discussion of rogue AIs is all about intelligence gone awry. Does anybody fear a control system that mis-estimates the corn crop? No, its about a malicious non-empathetic machine entity that coldly calculates how to defeat us. And that requires more that the current AI's are delivering.

The fears of machine superintelligence are based on the belief that true AI is just around a corner. After all, we’re so advanced and the progress is only accelerating, it’s probably a few years away, at most.

Analogously, public figures started warning us about climate change, so it must be just a few years before the earth is uninhabitable. It should be pretty obvious that if we start worrying about significant threats only a "few years" before they kill us, that will be as a rule too late.

Carbon-driven global warming ("the greenhouse effect") was gathering support in the 60s, followed by consensus in the 80s. Yet we are just now experiencing the first years of growth without increased carbon output.

The claim by the concerned parties with respect to harmful AI is that it may be a threat in the next 25 to 50 years: 2043 and 2065 are common estimates. "A few years" isn't a good way to characterize that, interpret Gates, etc., or a reasonable understanding of when we start worrying.

Current AI technologies give us no insight about the probability of an AGI

Because that is exactly the point. We can't look at what we have built today and extrapolate that AGI will come from it.

All anyone can do is speculate wildly on AGI just as they were able to in 1956 [1]. The difference is that we see automation/computing as ubiquitous (smartphones/computers in 2015), rather than around the edges (Radio/TV 1956). Which brings it closer to home in some ways.

[1] I chose 1956 because of the dartmouth conference on AI

Saying that we don't have to worry because AI development will take 1000 years is also missing the point. Research into unsafe AI is inherently faster than research into safe AI, because the latter problem is more constrained. If unsafe AI kills us all, the best time to start working on safe AI is 100 years ago, and the second best time is now.

Saying that we don't have to worry because smarter than human intelligences will have a better understanding of morality is also missing the point. All currently known mathematical formalisms for decision-making allow you to plug in arbitrary utility functions, including "immoral" ones. Thinking that future decision-making formalisms will support only "moral" utility functions is extreme wishful thinking.

Saying that we don't have to worry because brain emulations or human cognitive enhancement will come first is also missing the point. These things can be just as dangerous as AI (e.g. see Robin Hanson's descriptions of how mind uploading increases Darwinian pressures on everyone, including non-uploaded people), and can also lead to AI more quickly.

And so on.

If you want to feel safe and protected, no one can stop you. But you don't have any good arguments for feeling safe. I'm sorry, but you really don't. I've been on the lookout for such arguments for years, and I'd be happier if they existed.

I'm going to play Devil's Advocate on this.

While it is true that all known decision-making formulas allow for immoral utility functions, I think it's a flaw of current approaches, such as reinforcement learning, that they only allow nonspecific utility functions. I can't write down an AI program to maximize paperclips -- there's no way, in current formalisms, to specify what that means.

I think the FAI ideologues and the mainstream machine-learning community can both rally around the prospect of being able to specify a goal in terms other than reinforcement. Scientifically, that's an entirely sensible research question for the field of AI/ML to tackle, and it also constitutes a major step away from the scenarios in which a dangerous AGI winds up accidentally programmed with random, arbitrary goals that nobody actually wanted. After all, fundamental philosophical questions aside, a machine that flies out of control at random when turned on is simply a bad machine.

That said, it's indeed very hard to directly specify a utility function involving paperclips. If our universe were a Game of Life universe and we knew exactly which configuration corresponds to a paperclip, I'd be able to do that right now. But since we don't know the true laws of physics, the "hard way" involves encoding some kind of Solomonoff prior over all possible physical universes, and a rule for recognizing paperclips in each of them. That's kind of a tall order.

There's a shortcut that involves encoding a precise mathematical description of a human mind into an AI, and passing the buck by saying "the AI must maximize whatever function the human mind would output, given enough time to think". That would actually be straightforward to implement, if we had a description of a human mind that we were willing to trust. Unfortunately, the naive form of that approach immediately fails due to acausal blackmail. There's no obvious fix, but some folks are trying to devise non-obvious fixes, and I suppose it can be made to work in the long run.

Of course, after you can formulate a utility function in terms of a precise description of a human mind, the next step is reformulating it in terms of the outputs of an actually existing person, say some webcam videos and written instructions. You instruct the AI to infer the simplest program that would generate these outputs (using something like the Solomonoff prior again), and then proceed as in the previous step. That part has its own problems, but I expect that it can also be made to work.

The whole approach is kind of a long shot, but I hope that I've given a sense that the problem could be solvable by careful human effort on the scale of years. There are other approaches as well.

Or we just need an approach to AGI that gives us more conceptual abstraction than AIXI and its Solomonoff-based reasoning. Which we needed anyway, since AIXI_{tl} is "asymptotically optimal" with an additive constant larger than the remaining lifetime of the Earth.

Luckily, there's quite a large amount of bleeding-edge research into exactly that: learning "white-box" representations that are understandable and manipulable to human operators.

And I'm very skeptical that mathematical Platonism is useful for AI: "mathematical objects directly" do not exist in the real world, and it is very much real-world things on which we want our software to operate. "Conceptual abstraction" simply refers to a learning algorithm that possesses a representation of, for instance, a chair, that is not composed entirely of a concrete feature-set (visual edges, orientations, and colors) and can thus be deployed to generatively model chairs in general.

Computational cognitive science is working towards this sort of thing, and the results should start to hit the machine-learning community fairly soon.

Well, there's an influential minority that thinks mathematical objects are all that exists (Tegmark multiverse). I don't necessarily agree with them, but that's one way to rigorously define the domain for a utility function, in a way that is not obviously exploitable. Another way is to define utility in terms of an agent's perceptions, but that is exploitable by wireheading, and IMO that flaw is unfixable as the agent gets more powerful. I'm not aware of any other approaches that are different in principle from those two, so I'll stick with the lesser evil for now, and hope that someone comes up with a better idea.

I mean that is what I think, and most of the community thinks, but we haven't proven that empirically - nor could we, its unprovable.

But you don't have any good arguments for feeling safe.

Correct. Notice that I am not saying that we are safe. I personally think we aren't safe [1], but that is a different discussion which is equally as baseless.

The key here is that the argument that AGI will not be safe, and therefore needs to be regulated to make it safe, not only misses the point but ensures that we would never make one. The current level of funding for AGI development and understanding of the roadmap for AGI is absolutely dismal given it's potential impact. If we now add the constraint that in order to make one, we also ensure it doesn't threaten humanity - then you have sounded the death knell for AGI. Once we make AGI - in the sense that it is a recursively self improving system with human equivalent capabilities across domains - there is no way we will know what will happen. That's it.

[1] Which is why I am personally in the transhumanist camp.

I do agree that we lack evidence of a broad community of AI experts who have been able to make good predictions, which potentially puts AI research even behind economists! Nonetheless, we'd expect there to be a period between "wild guessing" and "good and reliable models"; Gates and Hawking either think we're in that middle ground, or they think the risks are considerable enough that it's worth pushing towards it now.

If I apply your analogy correctly, then warning about AI now, is the same as it would be warning about global warming in the 19th century. Not very logical and certainly very paranoid.

I don't see what's irrational about trying to solve problems before they become problems, rather than trying to do damage control after the fact.

Nothing in the world is more dangerous than premature attempts at safety.

But are you saying there's absolutely nothing that could've been done better in the industrial revolution? With some foresight we might have thought to develop solar power more urgently, for example, but no one thought we needed to. Worrying just a little might have changed that.

The point is not to stop progress, just to approach with caution and be mindful of what the long-term implications are.

And that, mind you, is still with the anachronistic application of 21st-century knowledge to the matter. If people in the 19th century had actually tried to figure out what they should be worrying about if they were going to worry two centuries prematurely, they would likely have come up with something completely different.

With some foresight, people were trying to develop solar, nuclear, and fusion all the way back in the '60s and '70s, but political interests ensured that research funding was reallocated towards shorter-term political projects.

It's akin to someone back then worrying about the long-term effects of steam/water vapor.

You don't know if vapor is actually a problem, or will become one, and you don't have any data in any field to back up your concerns.

When pharma companies produce a new drug which appears to have a positive effect, they don't know that it will cause problems and don't have any data to back up any concerns.

So do they rush full-steam-ahead into the unknown? Do you unleash the drug to whomever wants it, without regulation? No, of course not; you approach with caution, you hold clinical trials, you collect data as you go and you look for potential problems before the drug is widespread.

Arguably a similarly controlled approach would have helped in the case of carbon output and will help in the case of AI. It's not about fear mongering or preventing progress, it's about discussing ways to approach with caution and solve problems (and minimise the damage they cause), if and when they arise.

If someone can do forecasting of AGI with similar empiricism that would be revolutionary and welcomed.

Moravec's 70s papers are roughly comparable to Arrhenius's formula. The basic computational power predictions (a minor extension of Moore's law) still seem to be basically correct. There's a reasonable debate to be had about whether more interesting predictions made by Moravec/Kurzweil/etc. have been validated or not.

But AI is plausibly 50 years away from harming large numbers of people[1]. By that standard of "years until harm", global warming should have been first publicly discussed in the 1980s or 90s. That is a bit later than it happened, and later than some would like.

[1] If researchers develop models that say it's more like 100 or 500 years out, I think that would be great and very helpful. But several researchers have put out credible models which put a crisis in the 40s to 60s. Skepticism about their approaches that identifies specific flaws or produces improved models is good, much better than hand-waved dismissals.

> The fears of machine superintelligence are based on the belief that true AI is just around a corner. After all, we’re so advanced and the progress is only accelerating, it’s probably a few years away, at most.

Versus MIRI [1], quoting Nick Bostrom, whose book is arguably what most recently sparked all of the current discussion of AI risk:

> If what readers take away from language like “impending” and “soon” is that Bostrom is unusually confident that AGI will come early, or that Bostrom is confident we’ll build a general AI this century, then they’ll be getting the situation exactly backwards.

> [...]

> > My own view is that the median numbers reported in the expert survey do not have enough probability mass on later arrival dates. A 10% probability of HLMI [human-level machine intelligence] not having been developed by 2075 or even 2100 (after conditionalizing on “human scientific activity continuing without major negative disruption”) seems too low.

> > Historically, AI researchers have not had a strong record of being able to predict the rate of advances in their own field or the shape that such advances would take. On the one hand, some tasks, like chess playing, turned out to be achievable by means of surprisingly simple programs; and naysayers who claimed that machines would “never” be able to do this or that have repeatedly been proven wrong. On the other hand, the more typical errors among practitioners have been to underestimate the difficulties of getting a system to perform robustly on real-world tasks, and to overestimate the advantages of their own particular pet project or technique.

I agree with the meat of the article, though – recent AI progress has surely been oversold by the media. It's really frustrating to see people continually arguing against strawmen when there are some perfectly lovely arguments to grapple with instead!

Aside: is there some kind of law we could coin that if a rebuttal to AI risk starts off by referencing Skynet and Terminator, we can safely assume it's not going to be worth reading?

[1] https://intelligence.org/2015/01/08/brooks-searle-agi-voliti...

No, the fear of AGI is that the time between when we first start to see promising progess in AI to the point where AI is out of our control would be so short that we would not be able to react to prevent it.

That seems pretty absurd to me. A human-level AGI is going to require a sizable amount of time and data to train, so that it knows something rather than nothing. In fact, getting to the point where self-improvement becomes viable will take a good deal longer than training/educational periods for most other tasks. The information complexity is just higher.

So while AI safety should be taken seriously, we do need to "sober up" by remembering that an AGI has to obey the rules of information theory.

For instance if you include whole-brain emulations in AGI (and we should because the consequences are essentially the same) it would take zero time to train. But even without whole-brain emulations a child learns rapidly from its environment alone. An AGI could conceivably do the same but faster, and moreover could take advantage of access to the Internet.

I don't know what information theory has to do with any of that.

AGI has already been imagined. And yes, it does have to obey the laws of information theory: only one bit of knowledge can be learned from sensory inputs with one bit of entropy.

>An AGI could conceivably do the same but faster, and moreover could take advantage of access to the Internet.

Children still take 20 years to grow from infancy to being useful adults. If we hypothesize that an AGI could learn, say, 10x faster, we're assuming a two-year training time.

Even if you could somehow create a model precise enough to talk about information theory (define a symbol universe, map information to knowledge, itself a not-well defined term) it would not seem to be restrictive. Again, children take in a massive amount of sensory input.

Finally, the singularity hypothesis is that AGI could learn exponentially faster, not merely 10x. And again, WBEs don't need to learn at all.

However, I don't understand how the arguments correlate to "there is no need to worry about AI". The real worry should come from the fact that there are now many, many more people looking at the problem of AI than before, and it's a competition that now has serious money behind it.

The article mentioned the infinite monkey theorem as a counterargument, but ironically that is exactly how I think we will stumble upon generic intelligence. The more people that work on it and have the confidence, belief, and money to actually try out random things, the more likely it is that someone out there will discover the step change that will, in a day, take AI from deep prediction algorithms to eerily deliberate behavior.

Even if no one manages to shortcut our way to generic intelligence, we sure as hell will brute force it. While we're extremely early in the world of simulating organic brains, they exist. People have constructed virtual organic bodies controlled by virtual electric currents from virtual neurons in virtual brains. It's happening today, and as capacity, domain knowledge, and most importantly awareness increases, more and more people will work on the problem and bigger and bigger brains will be simulated.

I really don't see how we can question whether we will eventually reach human level intelligence running on computers. The only way I see generic machine intelligence not happening is by making people believe it is so impossible that no one ever tries to do it. And maybe that was the purpose of this article.

http://www.openworm.org/

The next thing you can do is optimize the time step element, so that for every 1 second of thinking in a silicon brain, it has performed the equivalent of 10 seconds of thinking in a human brain. At that point, you've surpassed human intelligence. It's also safe to assume that if we do reach this point, we've learned a lot more about how biological intelligence works, and may be able to use to our advantage the beneficial intelligence properties of savants.

The really big downside with all these approaches is that the intelligence will be built from a blueprint that includes emotions, and we may do something that is very morally wrong towards the intelligence that we created.

Why do people who are laypersons to machine learning continue to insist, despite what those in the field say, that we are learning nothing about the principles of general-domain learning from current developments? I think DeepMind is making genuine discoveries: they're certainly able to build things nobody was able to build before.

Do Any of those cited give specific timelines? Even if we are very far away, do you really doubt that one day machines will have superhuman intelligence? I take that as pretty much a given, whether it's 50 or 500 years from now. What I'm not so sure of is whether fear is an appropriate response.

Altman's bacteria handwashing analogy doesn't hold up. We don't care about bacteria because they have no central nervous system or consciousness on any level. However, we go out of our way to protect animals that can feel pain and experience emotions because it's what we've decided through our intelligence and higher reasoning is the moral thing to do. Stats show that the more intelligent and educated the human, the more likely he is to behave morally as our greatest moral philosophers define it. Why would super intelligent machines buck this trend?

We care about and protect some animals, yeah. However, we also industrially butcher hundreds of thousands of other animals every day to satisfy our goals.

Stats show that the more intelligent and educated the human, the more likely he is to behave morally as our greatest moral philosophers define it. Why would super intelligent machines buck this trend?

There's pretty large jump from "more intelligent => more moral" to "behave according to human morality and satisfy human goals".

Why? If you extrapolate from the amount of progress we have made toward AGI in the last 50 years (ie, none), then it's reasonable to argue that we still will have made no progress 50 and 500 years from now.

There are intellectual problems that humans aren't capable of solving; it wouldn't make any sense to talk about "superhuman intelligence" if that wasn't the case. The currently available evidence suggests that "constructing an AGI" might very well be one of those problems.

That's an odd way of defining progress.

> There are intellectual problems that humans aren't capable of solving; it wouldn't make any sense to talk about "superhuman intelligence" if that wasn't the case.

A superhuman intelligence doesn't necessarily have to come up with solutions humans would never think of, it just needs to come up with a solution in less time, or with less available data, or with fewer attempts.

It's probably safer to engineer the AI in such a way that it is guaranteed to be friendly than to trust that it will turn out that way and do nothing. Even a naive hedonistic calculus is probably safer than assuming that (human) ethics will result from more intelligence.

We don't regard bacteria like that, because they are not sufficiently intelligent, but because we believe they can't feel or suffer. So it's not a question of "level" of intelligence.

Whether artificial will is something that can be implemented in a turing machine and run on silicon seems an open question. I believe the concept, when deeply considered, is almost precisely antithetical to the goal of programming languages. The goal of artificial will is to give control to the program, not the programmer. Perhaps that's possible in a turing machine, but I have a feeling the "natural language" in which to express such a program would be like an inside-out LSD trip.

As to why this is the case you'd have to ask the researcher, but I think it's because the observation space would be too big for the machine running the agent (both memory and run time)

I guess Gates, Hawking and Stark are talking about a further AI creation, they are kind of long-term thinking guys.

It doesn't make sense to me personally why we need subjective experience, nor what it is in a physical sense. We can have agency and variability of choice without it, but at the same time we know how powerful it is in humans. Until we know exactly how it functions, i don't think anyone can claim with any certainty if it will happen, or how far away it is.

Imagine the public sentiment if mass media made people aware of the inevitability of the nuclear bomb 50 to 200 years in advance and people (not just the government) were actively working on developing it.

I think that had nuclear physics been more obvious 50-200 years earlier, it may have lead to a lot more practical private-sector development. In the case of power generation, this is seriously beneficial.

How will private-sector AI turn out? Will they boot up sentient AIs in Docker, then discard them? Is that OK? Who even knows?

We could probably have programmed a computer to drive a car back in the 70s, within an extremely specific configuration of roads and with no traffic or pedestrians. Now self driving cars have very advanced sensors, can cope with other traffic, pedestrians and a much wider range of road geometries. But it's still only useful for driving a car, you couldn't take the same program and teach it to even controll a boat or a plane let alone play Jeopardy, predict the weather or trade on the stock market. Systems like this are going to be very useful, but they are not taking us on a path to develop strong general AI.

And it's still only useful for driving a car in a rather specific configuration of roads and a somewhat limited set of conditions. I'd argue that a car's ability to operate autonomously and rather reliably under those circumstances is leading a lot of folks to be very optimistic about how long it will be before I can summon robo-taxi with my smartphone.

Everything else is parsing the sensor inputs in a language that makes sense to the AI answering machine.

So its knowledge category and context. Form a big enough and comprehensive graph of interconnected elements, and traversing that graph is what AI will do eventually. Apart from growing the graph and using the knowledge within the graph to improve itself.

What do people do? Advanced pattern-based behavior generation and unsupervised hierarchical spatial temporal pattern learning and abstraction. Logic and reasoning. Attention and goals.

I believe we will start to see somewhat convincing human-like conversational interface, speaking and movement from machines in lesx than five years. I don't even believe these things require real breakthroughs necessarily -- we can probably mostly combine existing techniques.

http://futureoflife.org/misc/open_letter#signatories

Project Cybersyn was a Chilean project from 1971–1973 (during the government of President Salvador Allende) aimed at constructing a distributed decision support system to aid in the management of the national economy. The project consisted of four modules: an economic simulator, custom software to check factory performance, an operations room, and a national network of telex machines that were linked to one mainframe computer.

[0]: http://en.wikipedia.org/wiki/Project_Cybersyn

The jokes will not last long.