[Core] Add default excludes for working_dir uploads

[crypdick]

Description

Automatically exclude common directories (.git, .venv, venv, pycache) when uploading working_dir in runtime environment packages.

At a minimum we need to exclude .git/ because unlike the others, nobody includes .git/ in .gitignore. This causes Ray to throw a ray.exceptions.RuntimeEnvSetupError if your .git dir is larger than 512 MiB.

I also updated the documentation in handling-dependencies.rst and improved the error message if the env exceeds the GCS_STORAGE_MAX_SIZE limit.

Related issues

N/A

Additional information

This PR pytorch/tutorials#3709 was failing to run because the PyTorch tutorials .git/ folder is huge.

[gemini-code-assist]

Code Review

This pull request introduces a helpful feature to automatically exclude common directories like .git and venv from working_dir uploads, preventing common errors with large repositories. The implementation is clean, and it's great to see that it's accompanied by thorough documentation updates and both unit and integration tests. My only suggestion is a minor improvement to the type hinting for better code clarity.

[iamjustinhsu]

Nice nice. Just some questions:

Can you clarify the following scenarios in these scenarios:

• excludes=[], .rayignore=["file.txt"]
• excludes=["file.txt"], .rayignore=[]
• excludes=["file.txt"], .rayignore=["file.txt"]
• excludes=["file.txt"], .rayignore=["file2.txt"]

I'm wondering

1. Why do have "excludes" when we had ".gitignore" previously?
2. Is the use-case necessary if users can specify venv, .git, pycache in their .rayignore file?

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[crypdick]

Thanks for the review @iamjustinhsu . Some thoughts:

1. the sources are unioned, so a file is excluded if it matches any source. So for your first 3 examples, file.txt & the default files are excluded, and in the last example file.txt & file2.txt & the default files are excluded.
2. the excludes param existed before this PR. It's a programmatic way to specify exclusions instead of having to edit static files (.gitignore, .rayignore). This PR just adds some default values.
3. so yes, users are able to set this behavior by manually creating .rayignore. This PR is about improving the default UX.

I don't think users should have to learn about .rayignore in order to use Ray for the first time. For example, I am in the process of submitting a bunch of tutorials to the official PyTorch docs, and if anyone tries to run them they will immediately get the RuntimeEnvSetupError error since pytorch/tutorials/.git/ is so large. .git/ is never included in .gitignore, so users would always have to create a .rayignore file in their repos to prevent this behavior.

If I may flip the question: why is it desirable for the to upload .git/, .venv, __pycache__/ to workers by default? I don't see any value in doing this, only downsides: extra overhead, and the potential for production workloads to break. Ray should just work.

[iamjustinhsu]

Got it

why is it desirable for the to upload .git/, .venv, pycache/

IMO it's not, although I could be wrong. This looks good to me then. I'm not the code owner so feel free to ping someone from core and docs team

[edoakes]

@crypdick makes sense to me, but my only hesitation is that this could be a breaking change and would be hard/confusing to debug.

Could we temporarily add a log message when one of the new default excludes is encountered? The log message can say "Directory 'foobar' is now ignored by default. To disable this behavior, ..."

[crypdick]

sg @edoakes I've added a temporary warning and a TODO for this to be removed in a few releases. I used log_once() to prevent this message being spammed.

[edoakes]

Thanks!