If you look at the source code for Sizzle, and know something about compilers and virtual machines, you'll see that the Expr object is modeled as as set of VM instructions. chunker is a regular expression that "tokenizes" the selector into sets of instructions. The appropriate VM instruction is then run for each chunk inside a loop in the Sizzle function.
That's my explanation for how these selectors are, in fact, a language that is compiled and executed.
Sorry, but if you knew what you are talking about, you would not call a parser with callbacks a compiler with VM. Maybe SAX parsers are also compilers and run some VM?
What does parsed CSS selector compile to? Where is that VM and what does it run? Bytecode?
Sizzle is just a parser what parses given selector into parts, gets matchin DOM elements, filters them and returns what matches. There is no compilation or virtual machines involved, just some string parsing, callbacks and DOM API.
That's my explanation for how these selectors are, in fact, a language that is compiled and executed.