In a recent interview [1], Hassabis (DeepMind founder) said they'd try training AlphaGo from scratch next, so it learns from first principles. Without the bootstrapping step of "learn from a database of human games", which introduce human prejudice.
As a Go player, I'm really excited to see what kind of play will come from that!
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
"What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play", Sussman said.
Minsky then shut his eyes.
"Why do you close your eyes?" Sussman asked his teacher.
"So that the room will be empty."
At that moment, Sussman was enlightened.
It sounds like the point of this story is to illustrate by analogy that starting from first principles is sometimes a silly way to approach a problem, and by extrapolation that it's a silly way to make an AI that plays Go well.
Making an AI that plays Go well is not (and has never been) the real goal. They're trying to learn how to build a AI that can solve any problem.
I don't think that's the point of the story. In the story, Sussman says that because the initial state of his net was randomized, it will "have no preconceptions". But that's not true. It still has "preconceptions", but randomly chosen ones. Because Sussman didn't know what they were, that didn't mean they didn't exist, any more than closing your eyes means the room is empty.
The Taoist concept of the uncarved block, referenced in the title of the koan, refers to naturalness and simplicity. I'm sure someone more expert than me can give a better explanation but it seems highly relevant to the idea of learning to play Go based only on the rules, rather than any human tradition of strategy.
The actual quote: "Sussman told Minsky that he was using a certain randomizing technique in his program because he didn't want the machine to have any preconceived notions" makes no indication it was a neural net ;)
Eric Raymond kinda butchered the Jargon File when he took over maintenance, so it wouldn't surprise me if some of the text there is invented. The original Jargon File does not contain any koans:
Well, yes, but it's still humans standing on the shoulders of other humans. Even though human players do memorize opening books, it stays in the family so to speak. Meanwhile a human player facing an AI engine is battling both the AI, and great human players of the past (who invented the openings).
Well, we can extend that to say the biological systems are self-assembled randomly and selected through evolutionary algorithms, starting from random molecules on the sea floor.
That's already a known method to transfer "knowledge" from one model to another. I should double-check before quoting a paper, but I think that this one talks about this (http://arxiv.org/abs/1503.02531).
You train many models. Then you "distill" their predictions into one model by using the multiple predictions (from many models) as targets (for the single model trained afterwards).
You're right to point out that humans don't do that.
I think it would be "cheating" if you train BetaGo on AlphaGo, for the purposes for doing that experiment. The goal would be to have some kind of "clean room" where people fumble around.
Of course, you can also run the other experiment to see how fast you can bootstrap BetaGo from AlphaGo. That's also interesting.
I'm pretty sure that the reinforcement learning algorithm they are using is guaranteed to converge. It just takes a very long time to train, and using human games probably sped it up.
As far as I know, using neural networks for function approximation destroys the various convergence guarantees available. NNs can easily diverge and have catastrophic forgetting, and this is one of the things that made them challenging to use in RL applications despite their power, and why one needs patches like experience replay and freezing the networks.
I believe the whole point of pretraining on reference policies, which a collection of "optimally" played human games is, is just avoidance of bad local optimum.
It can be a case that training and learning on just a learned policy is going to get you stuck in a local optimum that is of worse quality than the one with pretraining.
If they stored all of the AI played games their reference policy (the data) would be of extreme value. You could train a recurrent neural network, without any reinforcement learning, that you could probably run on a smartphone and beat all of the players. You wouldn't need a monte carlo search too.
There are algorithms [1] that have mathematical guarantees of achieving local optimality from reference policies that might not be optimal, and can even work better than the reference policy (experimentally) - assuming that the reference policy isn't optimal. The RNN trained with LOLS would make jointly local decisions over the whole game and each decision would guarantee that a minimization of future regret is being done. Local optimality mentioned here isn't finding a locally optimal model that approximates the strong reference policy, it means that it will find the locally optimal decisions (which piece to put where) without the need for search.
The problem is that for these algorithms you have to have a closely good reference policy, and given a small amount of human played Go games, reinforcement learning was the main algorithm instead, it allowed them to construct a huge number of meaningful games, from which their system learned, which allowed them to construct a huge number of more meaningful games, etc.
But, now when they have games that have a pretty good (AlphaGo is definitely playing on a superhuman level) reference policy, they can train the model based on that reference policy and they wouldn't need a search part of the algorithm at all.
The model would try to approximate the reference policy and would definitely be worse than AlphaGo real-search based policy, but it wouldn't be significantly worse (mathematical guarantee). The model is trained starting from a good player, and it tries to approximate the good player, on the other hand, reinforcement learning starts from an idiot player, and tries to become a good player, reinforcement learning is thus much much harder.
As a Go player, I'm really excited to see what kind of play will come from that!
[1] http://www.theverge.com/2016/3/10/11192774/demis-hassabis-in...