Skip to content

Add concurency to the partition writing routine#628

Open
gruuya wants to merge 1 commit intomainfrom
concurrent-partition-writes
Open

Add concurency to the partition writing routine#628
gruuya wants to merge 1 commit intomainfrom
concurrent-partition-writes

Conversation

@gruuya
Copy link
Contributor

@gruuya gruuya commented Aug 23, 2024

This PR introduces a new modality that allows for concurrent pulling and flushing into temp files from input plan's streams, while preserving the former sequential/non-concurrent modality as well (achieved by setting min_file_count_hint to None).

The actual speedup from this is quite modest:
image

@gruuya gruuya requested a review from mildbyte August 23, 2024 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants