Naive trend-extrapolation (assuming 1 synapse == 1 parameter) implies 17 iterations of Moore's law before we get human level AI, or some sort of ASIC, I suppose. This is what made me think human-level AI is a ways off.
Brain dead trend-extrapolation is a pretty good prediction tool, but I suppose my assumption that modern NNs aren't more efficient was sort of an arbitrary assumption that makes the exercise useless. Is there any work on this?
I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. These system are about as optimized for lip-reading and translation as bees are for their various tasks, so it wouldn't be surprise me if it turned out that they are similarly powerful per synapse to NNs.
A key assumption I don't share with Detmer is that we must match the brain's raw computational power to have a chance at human-level AI. From an evolutionary point of view, most of the brain's computations surely must be superfluous to intelligence and necessary instead to a vast host of other biological functions.
> Naive trend-extrapolation (assuming 1 synapse == 1 parameter) implies 17 iterations of Moore's law before we get human level AI, or some sort of ASIC, I suppose.
I'm not sure how you get that. My point is that artificial NNs may well be more efficient than biological ones because they are punching so far above their weight based on what you would expect from a naive comparison of parameters, in which case it's not 17 iterations but a lot less. Further, modern NNs don't typically max out the memory of a single GPU (since we want to do minibatches of at least n=10 for the training phase and researchers care a lot less about the forward passes or deployment), and groups like Google Brain or Deepmind have shown the ability to use ~1000 GPUs, so that's 4 orders of magnitude right there (doable with asychronous training like synthetic gradients).
> I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size.
What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
The human brain has 100 trillion synapses. So it would be 17 iterations of Moore's law for computing power to increase 100000x. That's what I was thinking. Wrong to assume NNs are no more efficient than brains. You know way more about this than me, so I'm just going to replace my opinion on this with your own. If you think we can chop off 4 orders without the help of Moore, I believe you, though this is much less comforting than my prediction!
>What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
Brain dead trend-extrapolation is a pretty good prediction tool, but I suppose my assumption that modern NNs aren't more efficient was sort of an arbitrary assumption that makes the exercise useless. Is there any work on this?
I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. These system are about as optimized for lip-reading and translation as bees are for their various tasks, so it wouldn't be surprise me if it turned out that they are similarly powerful per synapse to NNs.