Yeah Polars is an awesome library! I don't know a whole bunch about its internals, but I think it implements a bunch of additional speed ups through memory allocation and lazy evaluation, so its unlikely Pandas is will get to a similar speed without some huge changes elsewhere
Polars’ lazy evaluation is a big deal — this lets it do query plan optimization.
Whereas in Pandas every step is eager so it can’t look ahead to eliminate redundant steps. You basically can’t do a lot of query optimization in a multi step transform.
