GH-36889: [C++][Python] Fix duplicate CSV header when first batch is empty

[rynewang]

Rationale for this change

Fixes #36889

When writing CSV from a table where the first batch is empty, the header gets written twice:

table = pa.table({"col1": ["a", "b", "c"]})
combined = pa.concat_tables([table.schema.empty_table(), table])
write_csv(combined, buf)
# Result: "col1"\n"col1"\n"a"\n"b"\n"c"\n  <-- header appears twice

What changes are included in this PR?

The bug happens because:

1. Header is written to data_buffer_ and flushed during CSVWriterImpl initialization
2. The buffer is not cleared after flush
3. When the next batch is empty, TranslateMinimalBatch returns early without modifying data_buffer_
4. The write loop then writes data_buffer_ which still contains stale content

The fix introduces a WriteAndClearBuffer() helper that writes the buffer to sink and clears it. This helper is used in all write paths:

• WriteHeader()
• WriteRecordBatch()
• WriteTable()

This ensures the buffer is always clean after any flush, making it impossible for stale content to be written again.

Are these changes tested?

Yes. Added C++ tests in writer_test.cc and Python tests in test_csv.py:

• Empty batch at start of table
• Empty batch in middle of table

Are there any user-facing changes?

No API changes. This is a bug fix that prevents duplicate headers when writing CSV from tables with empty batches.

• GitHub Issue: [C++][Python] Duplicate csv header when table batches start with empty #36889

[github-actions]

⚠️ GitHub issue #36889 has been automatically assigned in GitHub to PR creator.

[rynewang]

@wgtmac Hi would you mind to also review this? Thanks!

[HuaHuaY]

I think we can clear data_buffer_ at last line of writeHeader. Relying on the Resize operation in TranslateMinimalBatch to perform implicit clear operation is not very readable.

[rynewang]

Clearing after writeHeader is a great idea! However it's not complete and failed my unit tests because there are other Write* methods. Now, I made a helper function and let all Write* to do "write to sink and clear buffer". @HuaHuaY can you help pointing this to a maintainer for approval? Thanks!

[HuaHuaY]

cc @pitrou @wgtmac . Please take a look.

[github-actions]

⚠️ GitHub issue #36889 has been automatically assigned in GitHub to PR creator.

[wgtmac]

Thanks for the update! This looks good to me now.

@pitrou Do you want to take a look at it?

[wgtmac]

Thanks @rynewang for fixing this and @HuaHuaY for the review!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 0d0e068.

There were 2 benchmark results indicating a performance regression:

• Commit Run on amd64-m5-4xlarge-linux at 2026-01-27 20:40:32Z
  • file-read (Python) with compression=uncompressed, dataset=nyctaxi_2010-01, file_type=feather, output_type=table
  • file-read (Python) with compression=uncompressed, dataset=fanniemae_2016Q4, file_type=feather, output_type=table

The full Conbench report has more details. It also includes information about 7 possible false positives for unstable benchmarks that are known to sometimes produce them.