I thought most Mozilla software was internally based on XPCOM. Then why can't they just proxy the interface calls to go to a different process via standard IPC mechanisms? (I've no knowledge of XUL extension model thou)
The issue is that extensions expect to access web content synchronously, so you need synchronous IPC for perfect compatibility, but that causes your UI process to block on your web process.
