Some good stuff here! I'm surprised they didn't show many analytical results in neural networks. For example, I like making candidates for research in deep learning derive back propagation. You can show a wide variety of interesting results in single neuron models as well.
In a lot of academic fields it's assumed that a researcher really understands their stuff to first principles. I think being able to derive back prop is a really straightforward exercise and definitely you should expect a researcher in the area to be able to do it off the cuff. I think it's akin to fizz buzz; it'll weed out people who really don't know what they're doing but it won't tell you too much about those who do it without trouble.
I thought so too, until my friend told me that CIT PhD and an applied research scientist in a FANNG company derived an incremental Gaussian mixture model without using the property of GMM at all, and another CIT PhD in the same team defended the algorithm by saying something like "but the intuition is correct". I couldn't believe my ears.
but most researchers don't use first principles day to day, like others in this thread -have said - if you don't use it you lose it. researchers don't utilize the details of back prop in there day to day work, so expecting an off the cuff derivation isn't a fair assessment of what makes a good researcher
Researchers specialize and as such even in a "single field" they work at very different levels of abstraction; one subfield will care about building up some novel construction from first principles and then another subfield will want to use that construction as a basic axiomatic building block and abstracting from the details. E.g. execution performance optimization of a known formula is orthogonal to developing a better formula, we want people working on both these aspects, these are going to be different people who each build on their own subfields first principles that don't overlap much with the first principles of the other researcher.
In my experience analytical results have very little impact on practical deep learning. Examples:
- Everyone “knows” that the Adam optimizer’s proof is incorrect, but we still use Adam because we don’t want to redo hyperparameter search with a different optimizer that’s proven to converge but probably performs worse.
- Everyone “knows” that the Wasserstein loss for GANs has a better convergence proof, but nobody uses it because the generated images look like crap compared to what you get from stylegan* with their default config.
It’d be nice if ML proofs led to better performance, but that’s not often the case. I see far more progress from better data preprocessing and from bringing in knowledge from other fields like signal processing.
that seems like a very basic thing that would be on a quals exam. a good researcher is a combination of smart/creative with expert knowledge of their field down to the fundamentals.
as an ee phd can you derive basic control theory or electromagnetic relationships?
