fix: prevent heap overflow in large monorepo scans#1038
Closed
fix: prevent heap overflow in large monorepo scans#1038
Conversation
Add streaming-based filtering to globWithGitIgnore to prevent heap overflow when scanning large monorepos with 100k+ files. Instead of accumulating all file paths and filtering afterwards, files are now filtered during streaming which dramatically reduces memory usage. Changes: - Add `filter` option to globWithGitIgnore for early filtering during streaming - Add createSupportedFilesFilter helper to create filter from supported files - Update getPackageFilesForScan to use streaming filter Fixes SMO-522 Ported from v1.x commit 9bbb8e8 ([SMO-522] Fix heap overflow in large monorepo scans #1026) Co-authored-by: Mikola Lysenko <mikolalysenko@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
createSupportedFilesFilter()function to create filter functions from supported file patternsfilteroption toGlobWithGitIgnoreOptionsfor streaming-based filteringglobWithGitIgnore()to apply filter during streaming instead of accumulating all files firstgetPackageFilesForScan()to use streaming filter to avoid OOM errors in large monorepos (100k+ files)Test plan
🤖 Generated with Claude Code
Note
Implements streaming-based file filtering to reduce memory usage during scans and updates dependencies.
filteroption toglobWithGitIgnore()and applies it during stream processing (supports negated gitignore patterns)createSupportedFilesFilter()and uses it ingetPackageFilesForScan()to filter supported files earlyexternal-tools.jsonto@coana-tech/cli@14.12.148CHANGELOG.mdwith Unreleased notes for the fixWritten by Cursor Bugbot for commit 3712e70. Configure here.