File Format API for PyIceberg

[nssalian]

Feature Request / Improvement

Problem

The write path in pyiceberg/io/pyarrow.py is hardcoded to Parquet. The write.format.default table property exists but is never read. Adding a new format (ORC, Vortex, Lance) requires modifying the monolithic write_file() function. The read path already dispatches multiple formats; the write path should too.

Proposal

Introduce a File Format API aligned with Java Iceberg's File Format API (design doc).

New module pyiceberg/io/fileformat.py:

• FileFormatWriter (ABC)
• FileFormatModel (ABC)
• FormatRegistry
• DataFileStatistics (it's in pyarrow.py currently but I think this might be good to consolidate for metrics)

Changes to pyiceberg/io/pyarrow.py:

• ParquetFormatWriter / ParquetFormatModel using the write_parquet() (inside write_file()
• write_file() refactored to read write.format.default, look up the format model, and dispatch.

TCK tests/io/test_file_format_tck.py:

• pytest-parameterized round-trip, statistics, type coverage, and null handling tests for every registered format.

Phased rollout:

• ABCs and registry first, then Parquet extraction with TCK tests, then write_file() dispatch

Java ↔ Python Mapping

Java	Python
FormatModel<D, S>	FileFormatModel (ABC, no type params)
FileAppender<D> / ModelWriteBuilder	FileFormatWriter (ABC)
FormatModelRegistry	FormatRegistry (keyed by FileFormat only)
Metrics	DataFileStatistics (existing)
TCK	test_file_format_tck.py

Scope

This proposal covers the abstraction layer and the Parquet extraction only. No new format writers are included; ORC write support (#20) and any future formats (Avro, etc.) would be follow-ups once this lands.

References

• Java File Format API: apache/iceberg#12774
• Design doc: Google Doc
• Format impls: Parquet #15253, ORC #15255, Avro #15254
• TCK: apache/iceberg#15415
• Prior pyiceberg ORC work: #20, #790, #2236

[rambleraptor]

This is going to be great, especially for new file formats!

I'm helping out with the TCK work over on Java and the shape of it might change a bit as we figure out exactly what we want the tests to look like. I don't think it would hurt for us to figure out our own testing story and then we can look into TCK when it's merged in.

[nssalian]

I'm helping out with the TCK work over on Java and the shape of it might change a bit as we figure out exactly what we want the tests to look like. I don't think it would hurt for us to figure out our own testing story and then we can look into TCK when it's merged in.

Sounds good @rambleraptor. The TCK can come in when ready. The first few pieces are critical.
CC: @kevinjqliu @Fokko @geruh to see if the proposal makes sense

[kevinjqliu]

Thanks for writing this up! This is super exciting. I think we can focus on both the read and write path for parquet. I'm interested to see what the API looks like.
We dont have to mirror the java apis, but perhaps we can reuse similar ideas

[kevinjqliu]

btw @pvary, you started a revolution 🥳

[nssalian]

Thanks @kevinjqliu . Could you assign this issue to me and I'll start pushing out the fix in phases.