To be honest I haven’t actually worked with any third party libraries that actually supported Pandas directly — most ML libs like Scikit require Numpy arrays (not Pandas dataframes) as inputs so I’ve always had to cast to Numpy from Pandas anyway.
But yes, Polars and DuckDB can easily cast to Pandas and also read Pandas dataframes in memory. I have some legacy data transformations that are mostly DuckDB but involve some intermediate steps in Pandas (because I didn’t want to rewrite them) and it’s all seamless (though not zero copy as it would be in a pure Arrow workflow).
And ironically DuckDB can query Pandas dataframes faster than Pandas itself due to its vectorized engine.
But yes, Polars and DuckDB can easily cast to Pandas and also read Pandas dataframes in memory. I have some legacy data transformations that are mostly DuckDB but involve some intermediate steps in Pandas (because I didn’t want to rewrite them) and it’s all seamless (though not zero copy as it would be in a pure Arrow workflow).
And ironically DuckDB can query Pandas dataframes faster than Pandas itself due to its vectorized engine.