which explains consciousness: both why we are self-aware (because we're modelling ourselves) and why we aren't more self-aware (because the model is only accurate enough to be effective)...

I think so, yep. (Although I would say that it explains our subjective experience of mind, "what it's like to be a human", rather than the existence and nature of subjective experience itself, but that's getting into metaphysics.)

It seems to me that there's no good reason not to reuse the term "ego" for the cybernetic self-models in human minds.

This idea also has interesting ramifications when you think about the boundaries of "self" and "other". The self-model system is not just the human being: it includes all the aspects and entities around the human too. And in humans we have models of other people and their self-models, and they have models of us and our self-models, and in turn we have models of their models of our models, etc.

FWIW I believe the other-models are what have been strongly selected for, and the self-model is just an epiphenomenon arising from the relative ease of modelling one's self after having gained the ability to represent models of others' selves:

cf https://news.ycombinator.com/item?id=23475069

Sounds reasonable to me.

I appreciate the link to Shannon's paper. I've played a browser-based version of such a machine/game and it's pretty eerie.

Long ago a friend of mine had this cat, she liked to sit on your lap, but hated it when you got up, she would always jump down the instant you started to rise.

However...

She could tell from the pattern of muscle tensions in your legs whether or not you were really getting up or just moving a bit to rearrange your butt or whatever, and she would only jump if you were getting up. She was flawless at this. In fact, a few times she jumped down a split-second before I knew I was getting up. Like Bruce Lee, she could detect and respond to the intention to move, even before the motion was consciously known to the mover himself (me, in this case.)