A lot of libraries depend on numpy directly. Unless polars is a drop in replacement (which I could be wrong but it doesn't seem like it is) then ultimately there's no avoiding pandas in many cases.
Numpy is mostly just a thin metadata wrapper around memory buffers. If pola.rs stores data in that format it should be able to give you zero-copy numpy objects pointing to the underlying pola.rs data.