With clause enhance #16353

shizy818 · 2025-09-05T02:45:09Z

Description

Content1 ...

Content2 ...

Content3 ...

This PR has:

Key changed/added classes (or packages if there are too many classes) in this PR

Copilot

Pull request overview

This PR enhances the WITH clause by adding support for CTE (Common Table Expression) materialization. The main objective is to optimize query performance by allowing CTEs to be materialized (executed once and cached) rather than being re-evaluated each time they're referenced. The implementation introduces a MATERIALIZED keyword, adds infrastructure for storing and reading CTE results, and integrates materialization into the query planning and execution pipeline.

Key changes include:

Added MATERIALIZED keyword support in SQL grammar for CTE declarations
Implemented CTE data storage infrastructure with memory buffering and configurable limits
Created CteScanNode and CteScanOperator for reading materialized CTE results
Enhanced query analysis and planning to handle CTE materialization
Added support for EXPLAIN and EXPLAIN ANALYZE with CTE queries

Reviewed changes

Copilot reviewed 68 out of 68 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
RelationalSql.g4	Added `MATERIALIZED?` keyword to namedQuery grammar rule
IoTDBConstant.java	Added constants for CTE query labels and folder naming
iotdb-system.properties.template	Added configuration properties for CTE buffer size and max rows
IoTDBConfig.java	Added CTE buffer configuration fields and getters/setters
IoTDBDescriptor.java	Added configuration loading logic for CTE buffer settings
WithQuery.java	Added materialized flag field to track CTE materialization intent
Query.java	Added materialized and done flags to track materialization state
ComparisonExpression.java	Made left/right fields mutable with setters for predicate reconstruction
LongLiteral.java	Changed parse method visibility to public for external access
CteDataStore.java	New class implementing CTE data storage with memory buffer and reference counting
CteDataReader.java	New interface defining contract for reading CTE data
MemoryReader.java	New class implementing in-memory CTE data reading
CteScanNode.java	New plan node representing a scan over materialized CTE data
CteScanOperator.java	New operator for executing CTE scans at runtime
CteMaterializer.java	Core logic for executing CTE queries and materializing results
PredicateWithUncorrelatedScalarSubqueryReconstructor.java	New class to optimize predicates by pre-executing uncorrelated scalar subqueries
Analysis.java	Added With clause tracking and named query map getter
StatementAnalyzer.java	Enhanced to set explain types and track CTE tables across scopes
Analyzer.java	Added CTE datastore registration from parent queries
Scope.java	Added table tracking to propagate CTE references through query scopes
ExpressionAnalyzer.java	Added subquery table tracking for CTE dependency resolution
RelationPlanner.java	Added materialized CTE processing logic with CteScanNode generation
SubqueryPlanner.java	Integrated uncorrelated scalar subquery folding for predicate pushdown
TableLogicalPlanner.java	Added CTE materialization invocation before query planning
Various optimizer files	Added CteScanNode handling in optimization rules
TableDistributedPlanner.java	Added explain result handling for CTE subqueries
AddExchangeNodes.java	Added CteScanNode distribution logic
TableOperatorGenerator.java	Added CteScanOperator generation
MPPQueryContext.java	Added CTE tracking fields and methods for materialization metadata
ExplainType.java	New enum to distinguish NONE, EXPLAIN, and EXPLAIN_ANALYZE modes
QueryExecution.java	Added CTE cleanup in resource release
ExplainAnalyzeOperator.java	Enhanced to merge CTE and main query explain analyze results
FragmentInstanceExecution.java	Updated statistics filtering to handle CTE virtual regions
FragmentInstanceStatisticsDrawer.java	Fixed fragment instance counting to exclude null data regions
DistributedQueryPlan.java	Added planText field for storing explain results
PlanVisitor.java	Added visitCteScan method
PlanGraphPrinter.java	Added CteScan and CorrelatedJoin rendering
TableModelStatementMemorySourceVisitor.java	Added CTE explain result merging logic
ConvertPredicateToTimeFilterVisitor.java	Enhanced to handle DoubleLiteral in time filters
Test files	Comprehensive test coverage including unit tests, integration tests, and matcher classes

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-08T01:37:19Z

...tion-test/src/test/java/org/apache/iotdb/relational/it/query/recent/IoTExplainAnalyzeIT.java

+      Pattern pattern = Pattern.compile(regex);
+      Matcher matcher = pattern.matcher(line);
+      if (matcher.find()) {
+        return Long.parseLong(matcher.group(1));


Potential uncaught 'java.lang.NumberFormatException'.

Suggested change

return Long.parseLong(matcher.group(1));

try {

return Long.parseLong(matcher.group(1));

} catch (NumberFormatException e) {

return 0;

}

Copilot · 2025-12-08T01:37:20Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java

+  }
+
+  public Map<Query, List<Identifier>> getSubQueryTables() {
+    return subQueryTables;


getSubQueryTables exposes the internal representation stored in field subQueryTables. The value may be modified after this call to getSubQueryTables.
getSubQueryTables exposes the internal representation stored in field subQueryTables. The value may be modified after this call to getSubQueryTables.

Suggested change

return subQueryTables;

return Collections.unmodifiableMap(subQueryTables);

Copilot · 2025-12-08T01:37:20Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java

+  }
+
+  public Map<NodeRef<Table>, CteDataStore> getCteDataStores() {
+    return cteDataStores;


getCteDataStores exposes the internal representation stored in field cteDataStores. The value may be modified after this call to getCteDataStores.

Suggested change

return cteDataStores;

return Collections.unmodifiableMap(cteDataStores);

Copilot · 2025-12-08T01:37:21Z

...e/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/Scope.java

+    return builder().like(this).build();
+  }
+
+  public List<Identifier> getTables() {


getTables exposes the internal representation stored in field tables. The value may be modified after this call to getTables.
getTables exposes the internal representation stored in field tables. The value may be modified after this call to getTables.

shizy818 · 2025-12-11T05:55:35Z

Sonar Coverage Report

Master branch: test coverage is 36.0%

with_clause_enhance branch: test coverage for new code is 80%

Copilot

Pull request overview

Copilot reviewed 67 out of 67 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

JackieTien97 · 2025-12-23T11:41:08Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java

+    // The buffer for cte materialization.
+    long cteBufferSizeInBytes =
+        Long.parseLong(
+            properties.getProperty(
+                "cte_buffer_size_in_bytes", Long.toString(conf.getCteBufferSize())));
+    if (cteBufferSizeInBytes > 0) {
+      conf.setCteBufferSize(cteBufferSizeInBytes);
+    }
+
+    // max number of rows for cte materialization
+    int maxRowsInCteBuffer =
+        Integer.parseInt(
+            properties.getProperty(
+                "max_rows_in_cte_buffer", Integer.toString(conf.getMaxRowsInCteBuffer())));
+    if (maxRowsInCteBuffer > 0) {
+      conf.setMaxRowsInCteBuffer(maxRowsInCteBuffer);
+    }
+


also add these in loadHotModifiedProps function, these two configurations should be hot loaded.

JackieTien97 · 2025-12-23T11:45:48Z

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template

+# will be used.
+# effectiveMode: hot_reload
+# Datatype: long
+cte_buffer_size_in_bytes=0


Suggested change

cte_buffer_size_in_bytes=0

# Privilege: SYSTEM

# unit: byte

cte_buffer_size_in_bytes=131072

JackieTien97 · 2025-12-23T11:46:36Z

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template

+
+# Max rows for CTE materialization
+# effectiveMode: hot_reload
+# Datatype: int


Suggested change

# Datatype: int

# Datatype: int

# Privilege: SYSTEM

JackieTien97 · 2025-12-23T11:58:40Z

.../main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceExecution.java

+    if (drivers == null || drivers.isEmpty()) {
+      return false;
+    }
+    // Check if any driver contains ExplainAnalyzeOperator
+    return drivers.stream()
+        .anyMatch(
+            driver ->
+                driver.getDriverContext().getOperatorContexts().stream()
+                    .anyMatch(
+                        operatorContext ->
+                            ExplainAnalyzeOperator.class
+                                .getSimpleName()
+                                .equals(operatorContext.getOperatorType())));


it should be inited as a field while generating this FragmentInstanceExecution in LocalExecutionPlanner instead of doing this judgement each time.

JackieTien97 · 2025-12-23T12:08:40Z

...va/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java

+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);


Suggested change

LOGGER.error("Fail to close fileChannel", e);

LOGGER.error("Fail to close CteDataReader", e);

JackieTien97 · 2025-12-24T07:20:58Z

...e/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteMaterializer.java

+            (tableRef, query) -> {
+              Table table = tableRef.getNode();
+              if (query.isMaterialized()) {
+                if (!query.isDone()) {


we may need another if to check whether this cte has already failed last time? if so we can just return even if this cte is not done

I thought even if it failed last time, maybe it's worthy try another time in case it's network problem.

JackieTien97 · 2025-12-24T07:31:19Z

...node/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/QueryPlanner.java

+    PredicateWithUncorrelatedScalarSubqueryReconstructor.getInstance()
+        .clearShadowExpression(predicate);


We shouldn't clear shadow expression here, the predicate expression is still being used afterwards. We can only clear them while front end finished.

JackieTien97 · 2025-12-24T07:32:43Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/CteDataReader.java

+
+import org.apache.tsfile.read.common.block.TsBlock;
+
+public interface CteDataReader {


Suggested change

public interface CteDataReader {

public interface CteDataReader extends Accountable {

JackieTien97 · 2025-12-24T07:32:55Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/CteDataReader.java

+   *
+   * @return the bytes used by this CteDataReader
+   */
+  long bytesUsed();


Suggested change

long bytesUsed();

long ramBytesUsed();

JackieTien97 · 2025-12-24T07:34:16Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/MemoryReader.java

+
+  @Override
+  public long bytesUsed() {
+    return INSTANCE_SIZE;


handle the memory footprint right.

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java # Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java # Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java

…ze predicate pushdown

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java # Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java

shizy818 force-pushed the with_clause_enhance branch from 8f4461d to 863c4c0 Compare September 17, 2025 02:23

shizy818 force-pushed the with_clause_enhance branch 2 times, most recently from caa8971 to 002f19a Compare September 29, 2025 06:07

shizy818 changed the title ~~With clause enhance (Review Only Now!)~~ With clause enhance Sep 30, 2025

shizy818 force-pushed the with_clause_enhance branch from 3e6aab7 to 1c1e3c7 Compare September 30, 2025 11:11

shizy818 force-pushed the with_clause_enhance branch from 1d2fbc6 to 0ce2a5d Compare October 21, 2025 06:23

shizy818 force-pushed the with_clause_enhance branch 2 times, most recently from 896c61b to 4f1233b Compare December 2, 2025 01:25

JackieTien97 requested a review from Copilot December 8, 2025 01:28

Copilot started reviewing on behalf of JackieTien97 December 8, 2025 01:29 View session

Copilot AI reviewed Dec 8, 2025

View reviewed changes

shizy818 force-pushed the with_clause_enhance branch from 208f2e9 to 2841855 Compare December 11, 2025 07:05

shizy818 force-pushed the with_clause_enhance branch 2 times, most recently from 7ac38e5 to 8c558c4 Compare December 23, 2025 01:31

JackieTien97 requested a review from Copilot December 23, 2025 11:36

Copilot started reviewing on behalf of JackieTien97 December 23, 2025 11:37 View session

Copilot AI reviewed Dec 23, 2025

View reviewed changes

JackieTien97 requested changes Dec 24, 2025

View reviewed changes

shizy818 and others added 11 commits December 26, 2025 14:28

cte materlization feat

9067b9a

spill data

5dfd94a

move cte data stores from Analysis to MPPQueryContext

68c7777

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java

remove spill-to-disk & add max_rows_in_cte_buffer config

8c44957

do not support CTE scan node serialization

b314d0c

execute uncorrelated scalar subquery in predicate in advance to utili…

cb41f41

…ze predicate pushdown

keep cte information in subquery

0515980

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java # Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java

fix bug

6f0d512

fix: duplicate CTE materialization process

ce51753

fix cteDataStore null issue

8489a87

shizy818 added 29 commits December 26, 2025 14:28

change default value for cte parameters

dff5ad8

fix unit test

1c8ba76

fix: CteScanNode replica set & query done

98b3942

fix: memory control & fetchCteQueryResult

d83f474

fix: with in cte materialization subquery

e1caa89

fix: with in scalar subquery

91629b7

fix: data store ref count

8397897

fix rebase

1503f79

mock executeForTableModel instead of fetchCteQueryResult

11f0169

add IT test for nest CTE

970aa48

add CteScanOperator test

f66fc88

fix sonar issues

ae9c504

fix explain & explain analyze output

2201ebe

add shadow expression for ComparisonExpression

3e2b7bd

fix: sorted original indices

936fcce

cteDataStores -> cteQueries

0f32812

remove cleanUpCTE

d390f52

simplify process

fcff595

copilot comments

57c2d65

revert getRetainedSizeInBytes

bfc9174

simplify CteScanOperator

5016903

claude ai: move ExplainType inside MPPQueryContext

e80713a

claude ai: MppQueryContext notation

c8c559d

fix: left & right in ComparisonExpression should be final

f102cbb

cleanup cte in main query

9f133c2

code review

e0f6bf0

PredicateWithUncorrelatedScalarSubqueryReconstructor part

63d49b4

materialize the query only once

22cfc09

memory for CteScanOperator / CteScanReader

fe47b8f

shizy818 force-pushed the with_clause_enhance branch from 4eb6aac to fe47b8f Compare December 26, 2025 06:30

	return subQueryTables;
	return Collections.unmodifiableMap(subQueryTables);

	return cteDataStores;
	return Collections.unmodifiableMap(cteDataStores);

-cte_buffer_size_in_bytes=0
+# Privilege: SYSTEM
+# unit: byte
+cte_buffer_size_in_bytes=131072

	LOGGER.error("Fail to close fileChannel", e);
	LOGGER.error("Fail to close CteDataReader", e);

		PredicateWithUncorrelatedScalarSubqueryReconstructor.getInstance()
		.clearShadowExpression(predicate);


		import org.apache.tsfile.read.common.block.TsBlock;

		public interface CteDataReader {

	public interface CteDataReader {
	public interface CteDataReader extends Accountable {

With clause enhance #16353

Are you sure you want to change the base?

With clause enhance #16353

Uh oh!

Conversation

shizy818 commented Sep 5, 2025

Description

Content1 ...

Content2 ...

Content3 ...

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

shizy818 commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shizy818 commented Dec 11, 2025 •

edited

Loading