make explicit without packages attached?

[e-kotov]

Hi @atsyplenkov , thanks for the awesome extension!

Do you think it would be possible to perform the "make explicit" operation without attaching the packages manually in the current R session?

[atsyplenkov]

Hi @e-kotov 👋

Thanks for your kind words. The short answer is yes. I have been thinking about this for a while, and it seems feasible to develop some kind of Rust CLI tool, with behaviour similar to cargo clippy or cargo fmt. It can be based on the astgrepr rules developed by @etiennebacher and look for libraries mentioned in DESCRIPTION or in renv.lock.

However, the main downside of this approach is that, since it has no access to the R session, it has no information about what packages have been attached directly in the R console, without mentioning library() in the R script. Moreover, the current implementation by @wurli relies on the base::search() function, which returns all attached packages, and the base::getNamespaceExports() function, which lists all functions from the attached packages. Without access to the R console, it may be tough to retrieve a list of exported variables from a namespace.

I have come up with three options for how it may potentially work. All of them rely on scanning scripts for library() calls (or similar with use() or pacman) in R scripts and returning an ordered list of used packages.

First, we can write a tool that will download source tarballs from CRAN and copy the NAMESPACE files. However, it may be computationally inefficient and relies on an internet connection. Additionally, the question is what to do with packages from GitHub or R-Universe? The presence of renv.lock can help with the latter, as it writes down the installation source, but it can still be tricky.

The second option is to start a separate R process in the directory from Rust and call getNamespaceExports() for all library() calls. For example, R -s -e "cat(getNamespaceExports('dplyr'), sep = '\n')". Again, it can take a while, but it can be run in parallel, and the results can be cached and linked to the package version/source. The problem is that R should be in the PATH and should be installed on the machine. The advantage is that it can call the required functions directly.

Last, but not least, is to get access to the ark kernel, as I believe it has to store this information. However, I have not explored the ark kernel much and have little understanding of how it works.

I am personally leaning towards the second option, as it seems the most stable and capable. I am happy to hear your thoughts and/or proposals. Cheers

[atsyplenkov]

Thinking about it further, I played around with option 2 today. I think I can come up with an MVP for Linux pretty soon... At least, I was able to reuse some bits of code from rig, and it works pretty well. I will make the repo public soonish.

As soon as the CLI tool will be ready, it would be just a matter of time writing a VS Code extension.

[etiennebacher]

Hi, just chiming in since I was mentioned in the issue (I don't plan to explore more on this), and I hope I understand this conversation correctly.

Do you think it would be possible to perform the "make explicit" operation without attaching the packages manually in the current R session?

The main problem here is that you can't know where the function comes from before you load the package. If I see a script with filter(), how can I know (without loading the session and the packages) whether it comes from stats or from dplyr? Sure, maybe I can scan the library() calls, but what if the script doesn't contain any library() call because it's called from another script that contains the library()?

The second option is to start a separate R process in the directory from Rust and call getNamespaceExports() for all library() calls. For example, R -s -e "cat(getNamespaceExports('dplyr'), sep = '\n')"

But this is not enough because you might load two packages that export the same function but if you don't interact directly with the user's session to determine which has precedence, then you can't reliably determine how to prefix the function.

For instance, collapse and dplyr both export replace_na(). If you first get the namespace of dplyr and then the one of collapse, you might determine that you should prefix the function with collapse, but maybe in the code dplyr is actually loaded second and therefore replace_na() is actually dplyr::replace_na().

This should also take into account explicit conflict declaration made with conflicted for instance, but that sounds more like an extra feature.

I hope this helps (curious to see what the MVP looks like implementation-wise).

[atsyplenkov]

Yeah, that’s a good observation. The conflicted packages can just raise a warning, like cargo clippy does.

[e-kotov]

But this is not enough because you might load two packages that export the same function but if you don't interact directly with the user's session to determine which has precedence, then you can't reliably determine how to prefix the function.

I have already stumbled upon this, and I understand there is no magical way even with LLM to guess exactly in some cases.

am happy to hear your thoughts and/or proposals

@atsyplenkov actually, having read the discussion and possible technical difficulties, I would say it is not such a big deal to spend so much time trying to make it work without packages attached.