I just want to point out: the history of software, just regular software, has been typified by the New Jersey approach and the MIT approach. The former consists in just hacking together something that kinda-mostly works, releasing fast, and trying to ameliorate problems later. The latter consists in thoroughly considering what the software needs to do, designing the code correctly the first time with all necessary functionality, (ideally) proving the code correct before releasing it, putting it through a very thorough QA process, and then releasing with a pre-made plan for dealing with any remaining bugs (that you didn't catch in the verification, testing, and other QA stages).
Only the latter is used for programming important things like airplanes, missile silos, and other things that can kill people and cost millions or billions of dollars when they go wrong -- the things in which society knows we cannot afford software errors.
I don't think it's even remotely a leap to say that we ought to apply only this thorough school of engineering to artificial general intelligence, whether it's at the level of an idiotic teenager or whether it's a superintelligent scifi thingy.
Now, if we think we can't really solve the inherent philosophical questions involved in making a "world optimization" AI that cannot go wrong, then we should be thinking about ways to write the software so that it simply doesn't do any "world optimization", but instead just performs a set task on behalf of its human owners, and does nothing else at all.
But either way, I think leaving it up to New Jersey kludging is a Very Bad Idea.
No one is making "world optimizations" engines. The concept doesn't even make sense when he wheels hit the road. No AI research done anytime in the foreseeable future would even be at risk of resulting in a runaway world optimizer, and contrary to exaggerated claims being made there would be plenty of clear signs something was amis if it did happen and planty of time to pull the plug.
I think you missed the last sentence: your software doesn't need to be a "runaway world optimizer" to be a very destructive machine merely because it's a bad machine that was put in an important job. Again: add up the financial and human cost of previous software bugs, and then extrapolate to consider the kind of problems we'll face when we're using deterministically buggy intelligent software instead of stochastically buggy human intellect.
At the very least, we have a clear research imperative to ensure that "AI", whatever we end up using that term to mean, "fails fuzzily" like a human being does: that a small mistake in programming or instructions only causes a small deviation from desired behavior.
Only the latter is used for programming important things like airplanes, missile silos, and other things that can kill people and cost millions or billions of dollars when they go wrong -- the things in which society knows we cannot afford software errors.
I don't think it's even remotely a leap to say that we ought to apply only this thorough school of engineering to artificial general intelligence, whether it's at the level of an idiotic teenager or whether it's a superintelligent scifi thingy.
Now, if we think we can't really solve the inherent philosophical questions involved in making a "world optimization" AI that cannot go wrong, then we should be thinking about ways to write the software so that it simply doesn't do any "world optimization", but instead just performs a set task on behalf of its human owners, and does nothing else at all.
But either way, I think leaving it up to New Jersey kludging is a Very Bad Idea.