No. But it was like for like. It was the one single SSH connection in both cases, simply running cacademo with and without a DISPLAY environment variable.
I think that part of the problem lies with SSH: the lack of batching as I mentioned, which might also be at ncurses' door too. Of course one wants terminal output to go over the wire as soon as possible, and not wait "until we have a buffer-ful". But not in this case.
Rather, batching up the control sequences to change colour, cursor position, and then output the relevant character into a single exchange over the wire would be better. That would be equivalent to a an X primitive to write character X at position Y with attributes Z, which presumably goes as a single message over the wire.
But a combination of whatever output buffering ncurses and libcaca are doing, what happens in the pseudo-terminal and sshd at the remote end, and then what happens at the local end when ssh is outputting to its pseudo-terminal, are all almost certainly strongly militating against that.
... In addition to converting lots of numbers from machine-readable to human-readable, then back to machine-readable again. (-:
I should mention that at one point I was transcoding from AIXTerm to ITU T.416 colour control sequences. The latter are about 4 times as long as the former, and that noticeably slowed things down even further. Yes, I'm in favour of 24-bit colour like the other terminal emulator authors, but it does come at a measurable price.