Skip to content

Wrap immutable plan parts into Arc #19893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
askalt wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Wrap immutable plan parts into Arc #19893

askalt wants to merge 1 commit into from
+266 −139

Conversation

@askalt
Copy link
Contributor

@askalt askalt commented Jan 20, 2026

Which issue does this PR close?

Rationale for this change

Improve performance of query planning and plan state re-set by making node clone cheap.

What changes are included in this PR?

  • Store projection as Option<Arc<[usize]>> instead of Option<Vec<usize>> in FilterExec, HashJoinExec, NestedLoopJoinExec.
  • Store exprs as Arc<[ProjectionExpr]> instead of Vec in ProjectionExprs.
  • Store arced aggregation, filter, group by expressions within AggregateExec.

@github-actions github-actions bot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate common Related to common crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jan 20, 2026
@askalt askalt force-pushed the askalt/arc_immutable branch from b9caf1d to bb97763 Compare January 20, 2026 08:17
@askalt askalt mentioned this pull request Jan 20, 2026
Open
@askalt
Copy link
Contributor Author

askalt commented Jan 20, 2026

run benchmark reset_plan_states

@alamb-ghbot
Copy link

alamb-ghbot commented Jan 20, 2026

🤖 Hi @askalt, thanks for the request (#19893 (comment)). scrape_comments.py only responds to whitelisted users. Allowed users: Dandandan, Omega359, adriangb, alamb, comphead, gabotechs, geoffreyclaude, klion26, rluvaton, xudong963, zhuqi-lucas.

@askalt
Copy link
Contributor Author

askalt commented Jan 20, 2026

@xudong963 could you please run benchmark reset_plan_states?

@askalt askalt mentioned this pull request Jan 20, 2026
Closed
@xudong963
Copy link
Member

xudong963 commented Jan 20, 2026

run benchmark reset_plan_states

@alamb-ghbot
Copy link

alamb-ghbot commented Jan 20, 2026

🤖 ./gh_compare_branch_bench.sh compare_branch_bench.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing askalt/arc_immutable (bb97763) to d90d074 diff
BENCH_NAME=reset_plan_states
BENCH_COMMAND=cargo bench --features=parquet --bench reset_plan_states
BENCH_FILTER=
BENCH_BRANCH_NAME=askalt_arc_immutable
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Jan 20, 2026

🤖: Benchmark completed

Details 

group     askalt_arc_immutable                   main
-----     --------------------                   ----
query1    1.00     32.2±1.09ms        ? ?/sec    1.01     32.4±0.31ms        ? ?/sec
query2    1.00      9.2±0.08ms        ? ?/sec    1.03      9.5±0.11ms        ? ?/sec
query3    1.00     12.2±0.25ms        ? ?/sec    1.15     14.1±0.08ms        ? ?/sec

@askalt
Wrap immutable plan parts into Arc
70a396a 
- Closes apache#19852

Improve performance of query planning and plan state re-set by making node clone cheap.

- Store projection as `Option<Arc<[usize]>>` instead of `Option<Vec<usize>>` in `FilterExec`,
  `HashJoinExec`, `NestedLoopJoinExec`.
- Store exprs as `Arc<[ProjectionExpr]>` instead of Vec in `ProjectionExprs`.
- Store arced aggregation, filter, group by expressions within `AggregateExec`.
@askalt askalt force-pushed the askalt/arc_immutable branch from bb97763 to 70a396a Compare January 20, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

common Related to common crate core Core DataFusion crate optimizer Optimizer rules physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate proto Related to proto crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Wrap more immutable plan parts into Arc

3 participants

@askalt @alamb-ghbot @xudong963