Hyperscan was recently posted.[1] A fast multi-pattern regex matcher. As in 1000's of patterns. Though optimized for deep packet inspection, I'm curious to try it for tokenizing.

[1] https://news.ycombinator.com/item?id=19270199