BUG: fixed_size_list pyarrow type not interpretable on round-trip write/read.

[delucchi-cmu]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import pyarrow as pa

a = pa.array([1, 2, 3])
b = pa.array([[True, False, True]] * 3, type=pa.list_(pa.bool_(), 3))
table = pa.Table.from_arrays([a, b], names=['a', 'b'])

pa_table = table.to_pandas(types_mapper=pd.ArrowDtype)
pa_table.to_parquet("using_types_mapper.parquet")

data_frame = pd.read_parquet("using_types_mapper.parquet")

Issue Description

In the example, we convert the PyArrow table to a pandas DataFrame, using the types mapper to enforce the fixed_size_list type of column b.

As of the versions indicated below, this results in the following error:

NotImplementedError: Passing pyarrow type specific parameters ([3]) in the string is not supported. Please construct an ArrowDtype object with a pyarrow_dtype instance with specific parameters.

If you inspect the generated file, you see that the pyarrow schema reflects the fixed_size_list. e.g pq.read_metadata("using_types_mapper.parquet").schema.to_arrow_schema().

If, instead, you do not indicate a types mapper, the type of fixed_size_list is lost, and the parquet file on-disk just uses a list type:

import pandas as pd
import pyarrow as pa

a = pa.array([1, 2, 3])
b = pa.array([[True, False, True]] * 3, type=pa.list_(pa.bool_(), 3))
table = pa.Table.from_arrays([a, b], names=['a', 'b'])

pa_table = table.to_pandas() # this is the different line
pa_table.to_parquet("loses_fixed_size.parquet")

data_frame = pd.read_parquet("loses_fixed_size.parquet")
display(pq.read_metadata("loses_fixed_size.parquet").schema.to_arrow_schema())

Expected Behavior

I would expect to

• read the parquet file without error
• for the returned dataframe to have some kind of list type for column b
• for the parquet file's on-disk metadata to still include a fixed_size_list type for column b

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.11.9.final.0
python-bits : 64
OS : Linux
OS-release : 6.5.0-1025-oem
Version : #26-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 12:35:22 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 2.1.1
pytz : 2024.1
dateutil : 2.9.0
setuptools : 72.1.0
pip : 24.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.27.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 17.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report. pyarrow is calling pandas_dtype with the argument fixed_size_list<item: bool>[3][pyarrow]. From the error message, it sounds like [3] should not be passed by PyArrow. @jorisvandenbossche - do you think this is on the PyArrow side?

Once this is resolved, it looks like this is a duplicate of #53011.

[jorisvandenbossche]

With the current infrastructure to preserve types on a parquet roundtrip (we store a string representation of the dtype in metadata, and then try to restore it by calling pandas_dtype(<string repr of the dtype>)), I would say this is on the pandas side to ensure it can roundtrip this.

Of course, it might not be possible to roundtrip a string representation for all dtypes, and it would be on the pyarrow side to suppress the error (if that is what we want).

This essentially boils down to:

In [23]: pd.api.types.pandas_dtype(str(pd.ArrowDtype(pa.list_(pa.bool_(), 3))))
---------------------------------------------------------------------------
...
File ~/scipy/repos/pandas/pandas/core/dtypes/dtypes.py:2352, in ArrowDtype.construct_from_string(cls, string)
   2348         except (NotImplementedError, ValueError):
   2349             # Fall through to raise with nice exception message below
   2350             pass
-> 2352         raise NotImplementedError(
   2353             "Passing pyarrow type specific parameters "
   2354             f"({has_parameters.group()}) in the string is not supported. "
   2355             "Please construct an ArrowDtype object with a pyarrow_dtype "
   2356             "instance with specific parameters."
   2357         ) from err
   2358     raise TypeError(f"'{base_type}' is not a valid pyarrow data type.") from err
   2359 return cls(pa_dtype)

NotImplementedError: Passing pyarrow type specific parameters ([3]) in the string is not supported. Please construct an ArrowDtype object with a pyarrow_dtype instance with specific parameters.