Add an xarray data source with ERA5 zarr example configurations

[bouweandela]

Description

Example usage:

Run

esmvaltool config copy data-gcs-era5.yml

or

esmvaltool config copy data-cds-era5.yml

to configure the data source. For the Climate Data Store, edit the file ~/.config/esmvaltool/data-cds-era5.yml to enter your CDS API key.

CMORized ERA5 data can then be loaded with:

In [1]: from esmvalcore.dataset import Dataset

In [2]: ds = Dataset(
   ...:     dataset="ERA5",
   ...:     project="native6",
   ...:     mip="E1hr",
   ...:     frequency="1hr",
   ...:     short_name="tas",
   ...: )

In [3]: ds.load()
Out[3]: <iris 'Cube' of air_temperature / (K) (time: 1323648; latitude: 721; longitude: 1440)>

or similar from a recipe. Inspiration on how to populate the variables key that maps between ERA5 and CMIP6 names can be found in recipe_daily_era5.yml.

Related to #2584

Link to documentation: https://esmvaltool--3179.org.readthedocs.build/projects/ESMValCore/en/3179/api/esmvalcore.io.xarray.html

Maybe we need to add the dependencies needed for various data sources as extras to avoid bloating our dependencies. Here gcsfs is needed to access data from Google Cloud Storage, but in #2917 we need xcube, and in #2690 we need intake-esm, etc.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

❌ Patch coverage is 60.81081% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.23%. Comparing base (583dcb7) to head (8112686).

Files with missing lines	Patch %	Lines
esmvalcore/io/xarray.py	60.81%	29 Missing ⚠️

❌ Your patch check has failed because the patch coverage (60.81%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3179      +/-   ##
==========================================
- Coverage   96.39%   96.23%   -0.17%     
==========================================
  Files         280      281       +1     
  Lines       16248    16322      +74     
==========================================
+ Hits        15663    15708      +45     
- Misses        585      614      +29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[BSchilperoort]

Thanks for adding this! Got some prototypes working with eWaterCycle forcing generation. The main limitation now is the available variables from the CDS, as Google's data has the wrong chunking strategy for our use case.

[bouweandela]

Thanks for testing @BSchilperoort!

It looks like (remote) zarr support will be available in Iris soon SciTools/iris#6961, so it may be possible to implement this without using Xarray.

Note to self, the following topics need further investigation:

• Can we add a configurable cache on local disk? https://zarr.readthedocs.io/en/stable/user-guide/experimental/#cachestore
• Do we need an additional configuration option to limit zarr library paralellism? https://zarr.readthedocs.io/en/stable/user-guide/performance/#using-zarr-with-dask
• Does using Xarray offer benefits over the plain iris implementation once it's available?
• Is there some way in which we can use facets to construct URLs, so we can use a single data source for multiple "files"? Related: can we enable facet discovery, perhaps by using fsspec to list available subpaths?