By "taking over the world", I meant establishing a Bostromian Singleton[1]. I didn't want to predict anything more, and certainly not anything resembling a human dictatorship.
I'm also not sure you could really limit your AI's reach. If for instance you give it a limited goal, like solving the Riemann Hypothesis, it could transform the solar system into a giant computer, killing off humanity along the way, just to solve the freaking mathematical problem.
Assuming you could limit the reach of your AI, you'd probably need to have a working Friendliness Theory to do that. At that point, I suggest you save the world instead.
I'm not sure that having a singleton is such a bad idea.
However, in your second scenario I highly doubt that an AI with transhuman reasoning would be unable to reason that (bear with me here) the reason it's solving said problem only exists because humans exist, and if they were killed off, it's reason for solving the problem would be redundant.
This doesn't mean that we can't limit the AI though, give it sufficient hardware and let it solve the problem (and don't converse with it or contact it in any matter unless it has solved the problem or requires more hardware), although this is like slavery and I don't like the concept, that would be one way of limiting it.
Oh I thought about your reasoning. It just need to keep the one that asked the question, then. Killing everyone else won't prevent it to tell the answer to the researcher.
Even with limited hardware, the AI still can have a non-trivial influence over the world, provided it can self-improve. And if you limit the hardware to present-day computers, and you ensure the format of the answer is highly constrained, you may be safe. The AI will be much more limited, though.
I'm also not sure you could really limit your AI's reach. If for instance you give it a limited goal, like solving the Riemann Hypothesis, it could transform the solar system into a giant computer, killing off humanity along the way, just to solve the freaking mathematical problem.
Assuming you could limit the reach of your AI, you'd probably need to have a working Friendliness Theory to do that. At that point, I suggest you save the world instead.
http://en.wikipedia.org/wiki/Singleton_%28global_governance%...