perf(core): optimize executor caching, conflict detection, and recursion tracking #33748
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Store the custom hasher when first retrieved instead of calling
getCustomHasher() twice per task - once to check if it exists and
again to use it. getCustomHasher traverses the executor lookup chain,
making this a significant optimization for large monorepos.
┌─────────────────────────────────────────────────────────────────────────┐
│ BEFORE: getCustomHasher called twice per task with custom hashers │
│ │
│ for (task of tasks) { │
│ const hasher = getCustomHasher(task, graph); // ← CALL 1 │
│ if (hasher) tasksWithCustom.push(task); │
│ } │
│ │
│ tasksWithCustom.map(async (task) => { │
│ const hasher = getCustomHasher(task, graph); // ← CALL 2 (REDUNDANT) │
│ await hasher(task, context); │
│ }); │
├─────────────────────────────────────────────────────────────────────────┤
│ AFTER: Store [task, hasher] tuple to reuse │
│ │
│ for (task of tasks) { │
│ const hasher = getCustomHasher(task, graph); // ← ONLY CALL │
│ if (hasher) tasksWithCustom.push([task, hasher]); // Store tuple │
│ } │
│ │
│ tasksWithCustom.map(async ([task, hasher]) => { │
│ await hasher(task, context); // ← Reuse cached hasher │
│ }); │
└─────────────────────────────────────────────────────────────────────────┘
Impact: 10-20% faster hash computation for monorepos with custom hashers
The npmPackageToPluginMap was being spread inside the package.json
detection loop, creating a new object copy for every single package.json
file processed. Now create the map once before the loop.
Also pre-compute Object.entries() to avoid repeated conversion.
┌─────────────────────────────────────────────────────────────────────────┐
│ BEFORE: Object spread on EVERY package.json (n times) │
│ │
│ for (file of packageJsonFiles) { │
│ const pluginMap = { │
│ ...npmPackageToPluginMap, // ← SPREAD 21+ entries every iteration! │
│ }; │
│ if (includeAngular) pluginMap['@angular/cli'] = '@nx/angular'; │
│ for ([dep, plugin] of Object.entries(pluginMap)) { ... } │
│ } │
├─────────────────────────────────────────────────────────────────────────┤
│ AFTER: Create map once, reuse for all files │
│ │
│ const pluginMap = includeAngular │
│ ? { ...npmPackageToPluginMap, '@angular/cli': '@nx/angular' } │
│ : npmPackageToPluginMap; // ← SPREAD ONCE (or zero times) │
│ const entries = Object.entries(pluginMap); // ← CONVERT ONCE │
│ │
│ for (file of packageJsonFiles) { │
│ for ([dep, plugin] of entries) { ... } // ← Reuse │
│ } │
└─────────────────────────────────────────────────────────────────────────┘
Impact: 15-30% faster plugin detection during nx init
Replace Object.keys().forEach() chains with for...in loops to avoid
creating intermediate arrays. This function is called for task graph
operations and benefits from avoiding array allocations.
┌─────────────────────────────────────────────────────────────────────────┐
│ BEFORE: Object.keys() creates arrays, forEach has callback overhead │
│ │
│ Object.keys(taskGraph.tasks).forEach((t) => { ... }); │
│ Object.keys(taskGraph.dependencies).forEach((taskId) => { │
│ taskGraph.dependencies[taskId].forEach((d) => { ... }); │
│ }); │
├─────────────────────────────────────────────────────────────────────────┤
│ AFTER: Direct iteration, no intermediate arrays │
│ │
│ for (const t in taskGraph.tasks) { ... } │
│ for (const taskId in taskGraph.dependencies) { │
│ for (const d of taskGraph.dependencies[taskId]) { ... } │
│ } │
└─────────────────────────────────────────────────────────────────────────┘
Impact: 10-15% faster reverse dependency calculation
…ion tracking - Cache getExecutorForTask results with WeakMap keyed by projectGraph - Optimize _getConflictingGeneratorGroups O(n²) to O(n) with generator-to-group Map - Use Set for O(1) lookup in dynamic-run-many startTasks - Use Set for O(1) visited tracking in operators.ts withDeps
👷 Deploy request for nx-docs pending review.Visit the deploys page to approve it
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This was referenced Dec 8, 2025
Collaborator
|
@adwait1290 We are very grateful for your enthusiasm to contribute, I kindly request that you please stop sending these AI assisted micro-perf PRs now. In future, please open an issue regarding your plans and do not simply send pages worth of micro PRs without open communication. Upon deeper inspection in some cases, we have found that they are not resulting in real-world performance wins, and instead create regressions because they are not considering memory and GC overhead of the whole system. We will work on better benchmarking infrastructure on our side to have greater confidence in CI as to whether these kinds of PRs are actually net wins but for now each individual PR requires a thorough investigation by the team and you are sending far, far too many. To reduce noise on the repo, I am going to close this, but rest assured it will be looked at as part of our performance optimization and benchmarking effort and merged in if it creates a provable net win. Thank you once again for your keenness to help make Nx the best it can be, we really appreciate it! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current Behavior
Several functions in the Nx core have suboptimal performance:
getExecutorForTask- Called multiple times for the same task without caching, repeatedly parsing executor files_getConflictingGeneratorGroups- O(n²) complexity searching through conflict groups withconflicts.find()+generatorsArray.some()startTasksin dynamic-run-many - Uses O(n)indexOf()per taskRowwithDepsin operators.ts - Uses O(n)indexOf()for visited tracking in recursionExpected Behavior
After this PR:
ASCII Diagram
Why Maintainers Should Accept This PR
getExecutorForTaskis called in task orchestration hot paths_getConflictingGeneratorGroupsaffects sync generator performancestartTasksis called for every task batch in dynamic terminal outputwithDepsis used when computing project graph subsetsRelated Issue(s)
Contributes to #33366
Merge Dependencies
Must be merged AFTER: #33738, #33742, #33745, #33747
This PR should be merged last in its dependency chain.