In my own lexer in JS, I considered using a generator but instead used a class with an index property. In your experience, what advantages does a generator have over that approach?
The big advantage (and possibly disadvantage) is that the class properties (i.e. the internal state of the iterator/lexer) are implicit, which means you don't need to worry about setting them and restoring them each time. For example, you could write something like this:
let idx = 0;
while (true) {
switch input[idx] {
case '"': {
const [token, length] = lexString(input.slice(idx));
yield token;
idx += length;
case '...': {}
}
}
Here, we don't need to worry about saving the state of the index, because it's implicitly "saved" at the yield, and we know we'll end up back on the line after the yield once control is returned to the generator.
In this case, where the only relevant state is the index, that's not that much of an advantage, but it could be more of an advantage if you were writing a generator function with more internal state - for example, a parser using a state machine to determine which tokens are valid next.
