Would be interesting to see this for various test functions with various starting parameters. Here is a large compilation of various test functions: http://infinity77.net/global_optimization/genindex.html

The functions in your link are not gradient based, not sure how they are applicable to neural networks with backprop.

Don't know what fraction of them, but most of them look differentiable (as long it is not exactly some variant of the Weierstrass function).

True, but I think these functions are intended for non-gradient based optimization algorithms.

Does someone want to take on the challenge of explaining the basics of this to a lay person? For starters, I don't understand the relationship between the scatter plot and the contour plot. I would guess that the blue (?) dots relate to the darker areas of the contour and the red dots to the lighter areas.

There are two classes in the scatter plot -- blue and red points. These can be separated by a line. A line has two parameters, for example, a slope and an intercept (although, other parameterizations are possible). These line parameters are the x and y axis. The lighter and darker areas of the heat map indicate better and worse lines, respectively, for separating the two classes of points.

That makes sense, thank you.

I was looking for a blogpost here that used similar, orange visualisation. The graphs seem to be an intuitively general application in geometry or analysis or whatever - I don't have the basics down; this should help.

What is the advantage of using Neopy instead that others neural network Python library like Pybrain?

There are a couple of advantages:

1. It's based on Theano, so it's fast. Also you are able to run your code on CPU and GPU.

2. NeuPy supports a lot of algorithms and different layer types (http://neupy.com/docs/cheatsheet.html), so you can easily construct deep neural networks.

3. In the NeuPy it's easy to read and understand network's structure. One of the newest features is a Subnetworks (http://neupy.com/docs/layers/basics.html#subnetworks).

4. Neural Network Surgery is another new feature (http://neupy.com/docs/surgery.html)