100% agreed. GEB is a great book, but Hofstadter hasn't been relevant in this field for a while now (AFAIK). Good old fashioned AI approaches show what a giant blind spot our minds are for our species, how little introspection goes on even within the brightest minds. Modern approaches to AI/ML have yielded tangible results and moved the field in the right direction, dismissing those results makes anyone look ridiculous in 2013.
Pure ML approaches are doomed to end up in a tar pit of AI mysticism. A statistical learning model can and will conclude that the thunder gods make the crops grow. People like Douglas Hofstadter and Eliezer Yudkowsky will have to use classical AI approaches to train our AI children in science and rationality.
A statistical learning model concluding that 'the thunder gods make the crops grow' would not make it a failure. Indeed that's what people thought for thousands of years!
I think you're confusing good AI with being 'smarter than humans'.
An AI can still pass the Turing test if it believes in Scientology or any of the other nonsesne beliefs that humans have.
That's fine if you want an AI hunter-gatherer. To have an AI scientist or engineer you need a rational thinking layer on top of the ML layer, so it can create hypotheses and try to falsify them. I predict the rational thinking layer will be built with good old fashioned AI: symbol processing, theorem proving, etc. Getting this to talk to the fuzzy ML layer will be a challenge.
My uninformed guess is that they mean that a statistical AI system might come to similar conclusions as humans such as concluding that there exist thunder gods, while a non statistical type (one with just "reason") would not come to that conclusion about thunder gods, and as such would not as accurately model humans.
Simple models of correlation and causation often make bizarre predictions, like the famous conclusion that pirates cause global warming. A statistical learning system cannot step outside its own predictions to fix such problems.
