Matrix calculus is a bit screwy when you realize that there are two possible notations to represent matrix derivatives (numerator vs. denominator layout; numerator layout is used in this guide). Plus, the notation is not very "speaking" for doing calculations unless you commit to memory some basic results.. which is why, as a physicist, I would recommend working in tensor calculus notation during calculations, and translating back to matrix notation for writing the results.
I was also surprised when I saw that there was no standard notation for Jacobian matrices. We use the numerator notation in the article, but point out that there are papers that use the denominator notation. I think I remember from engineering school that we used numerator notation so we stuck with that.
This is what index notation is good for, and I encourage everyone to learn it. Jacobians are dx_a/dx_b, two indices, and clearly b belongs to the derivative. Whether it's rows or columns is an implementation detail of how you're storing these numbers.
Index notation also seems natural for programming: an element A[i,j] or a slice Z[3,4,:] are precisely this.
I agree. If your Matrix calculus involves more complicated use of derivative operators you can't treat it like linear algebra anymore. Better to break it down into something like tensor notation first and back to matrices at the end. https://en.wikipedia.org/wiki/Del. Specifically I was always confused by the material derivative of a vector field when presented as either Matrix or vector calculus. If you represent it in tensor notation ( or explicitly break it out as operations on basis vectors ) it works out nicely.
