Inscrutable error when reading pyarrow string type columns in a dataframe

[vlulla]

Pandas recently, i think, got an option called future.infer_string, currently set to False, which will read the sequence of str objects as pyarrow string dtype. Since this is going to be the default in Pandas version 3.0, I have pandas.options.future.infer_string=True at the top of my script. With this option set i get an inscrutable error during some of my interactive sessions. A minimum reproducible example of my repl interaction is listed:

In [1]: import duckdb, pandas as pd
In [2]: s = ["string" for _ in range(5)]
In [3]: pd.options.future.infer_string = True ## changed to be prepared for upcoming update of pandas
In [4]: df = pd.DataFrame({"s":s})
In [5]: duckdb.sql("select * from df")
---------------------------------------------------------------------------
NotImplementedException                   Traceback (most recent call last)
Cell In[5], line 1
----> 1 duckdb.sql("select * from df")

NotImplementedException: Not implemented Error: Data type 'str' not recognized

In [6]: pd.options.future.infer_string = False ## default behavior
In [7]: df1 = pd.DataFrame({"s":s})
In [8]: duckdb.sql("select * from df1")
Out[8]:
┌─────────┐
│    s    │
│ varchar │
├─────────┤
│ string  │
│ string  │
│ string  │
│ string  │
│ string  │
└─────────┘
In [9]:

I have two questions:

1. Since pyarrow strings are going to become the default for collection of strings, is it possible to be make replacement scan not raise this error with str dtype?
2. Regardless of how this pandas dtype evolution is addressed, can this error message be improved? I would've never been able to distill this error had my table been bigger than just a few columns!

I love using duckdb from within the python repl and i believe that replacement scans are just so brilliant!

Thanks in advance!

[valankar]

pandas was just updated in my code and I started getting this error. I'm not yet sure what I need to do to fix.

[evertlammerts]

There is a PR up to fix this issue, and the fix will be in next week's bugfix release.

In the meantime I recommend downgrading Pandas to < 3.0.

[evertlammerts]

Duplicate of #37