Skip to content

Conversation

@jakenotjay
Copy link

@jakenotjay jakenotjay commented Feb 11, 2026

Motivation
Array.resize() enumerates all chunk coordinates if delete_outside_chunks is True (its default value), these old and new shapes are added into Python set objects to compute which chunks to delete. #3650 notes that this is totally unbounded in memory - even if the array is only growing and no chunks need deletion.

In our case our array has approximately ~220 million chunks, and we want to add a new time step. Building these two sets of tuples explodes memory to >20GB and does not complete (at least deployed in a no-swap environment, I never left my laptop running long enough). Furthermore, the set difference will always be empty because growing a dimension can't produce chunks outside the new shape.

Fix
Rather than always checking which is O(total_chunks), first check if the new_shape is >= the dimensions of the old shape. If array is only growing, skip enumeration.

This is a targeted mitigation, rather than a complete solution to the problem described by 3650, a more complete solution would construct outside chunk coords from the shape diff, rather than naively enumerating all coords.

TODO:

  • Add unit tests and/or doctests in docstrings
  • N/A Add docstrings and API docs for any new/modified user-facing classes and functions
  • N/A New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Feb 11, 2026
Copy link
Contributor

@rabernat rabernat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great fix, but it needs a test, no?

@jakenotjay jakenotjay requested a review from rabernat February 11, 2026 15:57
@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants