The bit about choosing good loss functions is a nice general point that I find is ignored in many machine learning set ups: if you're going to work so hard making a model optimize some objective, make sure it's the right objective!

Thanks. I (obviously) agree. The only problem is that it's not always computationally easy to optimize the thing you really care about.

Totally. It often becomes a trade-off of doing an alright job, e.g. approximately, optimizing the loss you're really interested in vs. doing a great job optimizing a tangentially related but much less appropriate loss.