Let LAYUP_CACHE_DIR set the default data location (#448)

[matthewholman]

Closes #448.

The problem

layup writes roughly 1.6 GB of SPICE kernels, observatory codes and debiasing tables to the directory pooch.os_cache("layup") picks, which lives under the user's home directory. On a cluster the home directory is frequently a different — and smaller — partition from the one layup is installed on, so that is the wrong place for it. Raised by @hannorein in #443.

A cache_dir argument already overrode this per call, and layup bootstrap has --cache. Neither addresses the issue, which is that the default should be settable once rather than threaded through every call site.

The change

Adds layup.utilities.cache_location.default_cache_dir():

export LAYUP_CACHE_DIR=/data/shared/layup-cache

and routes all ten call sites across eight files through it. Precedence is now an explicit cache_dir argument, then the environment variable, then the OS cache — so nothing that worked before changes.

Four details that were decided rather than defaulted:

• A blank or whitespace-only value falls back to the OS cache rather than writing into the current working directory. export LAYUP_CACHE_DIR= is a shell accident, not a request to scatter 1.6 GB wherever you happened to be standing.
• A leading ~ is expanded. An environment variable is not shell-expanded when a process reads it, so LAYUP_CACHE_DIR=~/data would otherwise create a directory literally named ~.
• Asking where the cache is does not create it. pooch still creates it on download, which is the previous behaviour.
• import pooch is dropped from the six files that only imported it for this.

Verification

Tested through the CLI, not just the helper — with the variable set, layup bootstrap --help reports the overridden path as its default; without it, the original:

$ layup bootstrap --help | grep -A4 -- "--cache CACHE,"
                        /Users/…/Library/Caches/layup)

$ LAYUP_CACHE_DIR=/data/shared/layup-cache layup bootstrap --help | grep -A4 -- "--cache CACHE,"
                        /data/shared/layup-cache)

Seven tests cover the four cases, including that the unset default is byte-for-byte what layup used before this change.

Full suite: 484 passed.

One thing worth knowing for anyone testing this

Three tests (test_comet_output, test_predict_output, test_get_onsky_data_output) appear to fail unless the layup under test is first on PATH. That is not a flake and not related to this change: layup dispatches by scanning PATH for layup-<verb> executables and shelling out to them, so any second installation silently wins over the one you just built. It cost me some time here, so it seems worth writing down. With PATH set correctly all three pass.

That mechanism might deserve its own issue — a CLI that resolves its own subcommands through PATH will pick up a stale sibling install without saying so — but it is out of scope here.

[mschwamb]

I think it would be helpful for this to be condensed into a shorter comment this is easier to understand. It references tickets when I think in 1-2 years will be confusing.

[mschwamb]

Maybe this won't happen but if I had one job running on an old cache and wanted to try a new one if they were running in the same environment the caches would get crossed? Maybe that's okay but maybe somewhere that needs to be mentioned in the documentation?

[matthewholman]

Both addressed in 456e84e.

On the comment length and the ticket references — you were right, and the second half especially. Issue numbers are shorthand for whoever is in the thread this week; a year from now they are a lookup someone has to do to read a docstring. It is now about half the length, says what the variable does and what the precedence is, and cites no issues.

On two jobs crossing caches — worth answering precisely rather than waving at it, because the answer is "mostly fine, except in exactly the case you described":

• The variable is read on each call, not captured at import, so a process can change it mid-run.
• Two processes with different values are independent.
• But it is a process setting inherited from the shell, so two jobs launched from the same shell share whatever that shell exports — which is your scenario. Setting it per job rather than exporting it once keeps them separate.

That is now in the module docstring, and per your suggestion in the documentation: the bootstrap section of the landing page gains the export line, notes that every command reads it, and carries a note about the shared-shell case.

The branch also merges main, so it now has the landing page from #478 to document this on.