diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml index 3342d752..29e48cb9 100644 --- a/.github/workflows/python-app.yml +++ b/.github/workflows/python-app.yml @@ -40,7 +40,7 @@ jobs: pip install -e packages/bundled_models/fourcastnext pip install -e .[test] - pytest -m="not noci" --cov=packages/data --cov=packages/utils --cov=packages/pipeline --cov=packages/training --cov=packages/zoo --cov=packages/bundled_models/fourcastnext --ignore=packages/nci_site_archive + pytest -m="not noci" --cov=packages/data --cov=packages/utils --cov=packages/pipeline --cov=packages/training --cov=packages/zoo --cov=packages/bundled_models/fourcastnext --cov=packages/third_party/pangeo_site_archive --ignore=packages/nci_site_archive - name: Coveralls GitHub Action uses: coverallsapp/github-action@v2.3.0 with: diff --git a/docs/data.md b/docs/data.md index f5a3cbb8..8e15c3d8 100644 --- a/docs/data.md +++ b/docs/data.md @@ -40,6 +40,20 @@ The package `site_archive_nci` (which is present in some of the tutorials) is th Additionally, you can explore the [geonetwork](https://geonetwork.nci.org.au/geonetwork/srv/eng/catalog.search#/home) to find additional data sets of interest which you might like to utilise by writing a custom PyEarthTools data accessor. +### Accessing CMIP6 climate model data from the cloud + +The package `site_archive_pangeo` (in `packages/third_party/pangeo_site_archive`) reads CMIP6 model output from the [Pangeo CMIP6 cloud archive](https://pangeo-data.github.io/pangeo-cmip6-cloud/accessing_data.html) on Google Cloud. No account, download or HPC access is needed; the data is opened lazily, so only the times you request are transferred. + +```python +import pyearthtools.data +from site_archive_pangeo import PangeoCMIP6 + +index = PangeoCMIP6(["tas", "pr"], source_id="CanESM5", experiment_id="historical") +ds = index.retrieve(pyearthtools.data.Petdt("2000-01")) +``` + +Each variable is looked up in the Pangeo catalogue and the newest version is used. If a variable exists on several grids, pass `grid_label` to choose one. Scalar coordinates that differ between variables (for example `height`, which is 2 m for `tas` but 10 m for `uas`) are dropped when the variables are combined. + ### Connecting PyEarthTools to a new dataset in any HPC facility by coding a new data accessor For on-disk data access, you will need to create a new accessor based on the [`pyearthtools.data.ArchiveIndex`](project:./api/data/data_api.md#pyearthtools.data.indexes.ArchiveIndex) class (or [`pyearthtools.data.Index`](project:./api/data/data_api.md#pyearthtools.data.indexes.Index) for some use cases). Additional instructions for this still need to be written. In the meantime, refer to the [NCI site archive source code](https://github.com/ACCESS-Community-Hub/PyEarthTools/tree/develop/packages/nci_site_archive) for examples. diff --git a/docs/notebooks/Gallery.ipynb b/docs/notebooks/Gallery.ipynb index f56b6e2a..4f706bd5 100644 --- a/docs/notebooks/Gallery.ipynb +++ b/docs/notebooks/Gallery.ipynb @@ -176,7 +176,8 @@ "| Transform-on-load | How to transform and adjust data at load-time | [Using Built-In Operations](./data/UsingTheInBuiltOperations.ipynb) | 18 Aug 2025 |\n", "| Applying data transforms | -- | [Applying Data Transforms](./data/Transforms.ipynb) | 18 Aug 2025 |\n", "| Geospatial subsetting | -- | [Region Cutting](./data/RegionCutting.ipynb) | 18 Aug 2025 |\n", - "| Remote Zarr Loading | Demontsrates loading data in a remote zarr archive | [Remote Zarr Loading](./data/Remote_Zarr_Loading.ipynb) |2 June 2026 |\n" + "| Remote Zarr Loading | Demonstrates loading data in a remote zarr archive | [Remote Zarr Loading](./data/Remote_Zarr_Loading.ipynb) |2 June 2026 |\n", + "| Pangeo CMIP6 | Demonstrates loading CMIP6 climate model data from the Pangeo cloud archive | [Pangeo CMIP6](./data/Pangeo_CMIP6.ipynb) | 11 Oct 2026 |\n" ] }, { diff --git a/docs/notebooks/data/Pangeo_CMIP6.ipynb b/docs/notebooks/data/Pangeo_CMIP6.ipynb new file mode 100644 index 00000000..96153724 --- /dev/null +++ b/docs/notebooks/data/Pangeo_CMIP6.ipynb @@ -0,0 +1,4456 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1000001", + "metadata": {}, + "source": [ + "# Accessing CMIP6 data from the Pangeo cloud archive\n", + "\n", + "The [Pangeo CMIP6 cloud archive](https://pangeo-data.github.io/pangeo-cmip6-cloud/accessing_data.html) hosts CMIP6 climate model output as analysis-ready, cloud-optimised (ARCO) Zarr stores on Google Cloud Storage. No account, download or HPC access is needed: PyEarthTools opens the stores lazily, so only the data you use is transferred.\n", + "\n", + "The `PangeoCMIP6` accessor, in the `site_archive_pangeo` package, is derived from the PyEarthTools `ArchiveIndex` class. It looks up each variable in the Pangeo catalogue and opens its Zarr store, so it is used like any other PyEarthTools data accessor.\n", + "\n", + "Install the package from the monorepo with `pip install -e packages/third_party/pangeo_site_archive`." + ] + }, + { + "cell_type": "markdown", + "id": "a1000002", + "metadata": {}, + "source": [ + "### Set up imports for demo" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "a1000003", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T18:32:47.666679Z", + "iopub.status.busy": "2026-10-14T18:32:47.665681Z", + "iopub.status.idle": "2026-10-14T18:32:52.808611Z", + "shell.execute_reply": "2026-10-14T18:32:52.807583Z" + } + }, + "outputs": [], + "source": [ + "import matplotlib.pyplot\n", + "import cartopy.crs\n", + "import numpy\n", + "\n", + "import pyearthtools.data\n", + "import pyearthtools.data.transforms.coordinates\n", + "import pyearthtools.pipeline\n", + "from site_archive_pangeo import PangeoCMIP6" + ] + }, + { + "cell_type": "markdown", + "id": "a1000004", + "metadata": {}, + "source": [ + "## Discover the available data\n", + "\n", + "The catalogue lists every variable of every model run in the archive. `PangeoCMIP6.catalog()` returns it as a pandas `DataFrame`, so the usual pandas filtering shows what is available. The catalogue is a large CSV file, so the first read takes a moment." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "a1000005", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T18:32:52.812756Z", + "iopub.status.busy": "2026-10-14T18:32:52.812756Z", + "iopub.status.idle": "2026-10-14T18:33:28.916808Z", + "shell.execute_reply": "2026-10-14T18:33:28.915794Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
| \n", + " | source_id | \n", + "experiment_id | \n", + "member_id | \n", + "table_id | \n", + "variable_id | \n", + "grid_label | \n", + "zstore | \n", + "version | \n", + "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "CMCC-CM2-HR4 | \n", + "highresSST-present | \n", + "r1i1p1f1 | \n", + "Amon | \n", + "ps | \n", + "gn | \n", + "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", + "20170706 | \n", + "
| 1 | \n", + "CMCC-CM2-HR4 | \n", + "highresSST-present | \n", + "r1i1p1f1 | \n", + "Amon | \n", + "rsds | \n", + "gn | \n", + "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", + "20170706 | \n", + "
| 2 | \n", + "CMCC-CM2-HR4 | \n", + "highresSST-present | \n", + "r1i1p1f1 | \n", + "Amon | \n", + "rlus | \n", + "gn | \n", + "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", + "20170706 | \n", + "
| 3 | \n", + "CMCC-CM2-HR4 | \n", + "highresSST-present | \n", + "r1i1p1f1 | \n", + "Amon | \n", + "rlds | \n", + "gn | \n", + "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", + "20170706 | \n", + "
| 4 | \n", + "CMCC-CM2-HR4 | \n", + "highresSST-present | \n", + "r1i1p1f1 | \n", + "Amon | \n", + "psl | \n", + "gn | \n", + "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", + "20170706 | \n", + "
PangeoCMIP6\n",
+ "\tDescription Pangeo CMIP6 analysis-ready, cloud-optimised data accessor\n",
+ "\t\t Documentation 'https://pangeo-data.github.io/pangeo-cmip6-cloud/accessing_data.html'\n",
+ "\n",
+ "\n",
+ "\tInitialisation \n",
+ "\t\t catalog 'https://storage.googleapis.com/cmip6/pangeo-cmip6.csv'\n",
+ "\t\t experiment_id 'historical'\n",
+ "\t\t grid_label None\n",
+ "\t\t member_id 'r1i1p1f1'\n",
+ "\t\t source_id 'CanESM5'\n",
+ "\t\t table_id 'Amon'\n",
+ "\t\t variables ['tas']\n",
+ "\tTransforms \n",
+ "\t\t StandardCoordinateNames {'latitude': "['lat', 'Latitude', 'yt_ocean', 'yt']", 'longitude': "['lon', 'Longitude', 'xt_ocean', 'xt']", 'replacement_dictionary': 'None', 'time': "['Time']"}\n",
+ "\t\t Trim {'__args': '()', 'variables': "['tas']"}<xarray.Dataset> Size: 34kB\n", + "Dimensions: (time: 1, latitude: 64, longitude: 128)\n", + "Coordinates:\n", + " * time (time) object 8B 2000-01-16 12:00:00\n", + " * latitude (latitude) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86\n", + " * longitude (longitude) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2\n", + " height float64 8B ...\n", + "Data variables:\n", + " tas (time, latitude, longitude) float32 33kB dask.array<chunksize=(1, 64, 128), meta=np.ndarray>\n", + "Attributes: (12/54)\n", + " CCCma_model_hash: 3dedf95315d603326fde4f5340dc0519d80d10c0\n", + " CCCma_parent_runid: rc3-pictrl\n", + " CCCma_pycmor_hash: 33c30511acc319a98240633965a04ca99c26427e\n", + " CCCma_runid: rc3.1-his01\n", + " Conventions: CF-1.7 CMIP-6.2\n", + " YMDH_branch_time_in_child: 1850:01:01:00\n", + " ... ...\n", + " title: CanESM5 output prepared for CMIP6\n", + " tracking_id: hdl:21.14100/872062df-acae-499b-aa0f-9eaca76...\n", + " variable_id: tas\n", + " variant_label: r1i1p1f1\n", + " version: v20190429\n", + " status: 2019-10-25;created;by nhn2@columbia.edu
Pipeline\n",
+ "\tDescription `pyearthtools.pipeline` Data Pipeline\n",
+ "\n",
+ "\n",
+ "\tInitialisation \n",
+ "\t\t exceptions_to_ignore None\n",
+ "\t\t iterator None\n",
+ "\t\t max_exception_count -1\n",
+ "\t\t name None\n",
+ "\t\t sampler None\n",
+ "\tSteps \n",
+ "\t\t pangeo_cmip6.PangeoCMIP6 {'PangeoCMIP6': {'catalog': "'https://storage.googleapis.com/cmip6/pangeo-cmip6.csv'", 'experiment_id': "'historical'", 'grid_label': 'None', 'member_id': "'r1i1p1f1'", 'source_id': "'CanESM5'", 'table_id': "'Amon'", 'variables': "['tas']"}}\n",
+ "\t\t __main__.ConvertUnits {'ConvertUnits': {'var_convert_dict': {'tas': '<function <lambda> at 0x000001A7004BBC40>'}, 'var_units': {'tas': "'degC'"}}}<xarray.Dataset> Size: 34kB\n", + "Dimensions: (latitude: 64, longitude: 128, time: 1)\n", + "Coordinates:\n", + " * latitude (latitude) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86\n", + " * longitude (longitude) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2\n", + " * time (time) object 8B 2000-01-16 12:00:00\n", + " height float64 8B ...\n", + "Data variables:\n", + " tas (time, latitude, longitude) float32 33kB dask.array<chunksize=(1, 64, 128), meta=np.ndarray>\n", + "Attributes: (12/54)\n", + " CCCma_model_hash: 3dedf95315d603326fde4f5340dc0519d80d10c0\n", + " CCCma_parent_runid: rc3-pictrl\n", + " CCCma_pycmor_hash: 33c30511acc319a98240633965a04ca99c26427e\n", + " CCCma_runid: rc3.1-his01\n", + " Conventions: CF-1.7 CMIP-6.2\n", + " YMDH_branch_time_in_child: 1850:01:01:00\n", + " ... ...\n", + " title: CanESM5 output prepared for CMIP6\n", + " tracking_id: hdl:21.14100/872062df-acae-499b-aa0f-9eaca76...\n", + " variable_id: tas\n", + " variant_label: r1i1p1f1\n", + " version: v20190429\n", + " status: 2019-10-25;created;by nhn2@columbia.edu
<xarray.Dataset> Size: 34kB\n", + "Dimensions: (latitude: 64, longitude: 128, time: 1)\n", + "Coordinates:\n", + " * latitude (latitude) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86\n", + " * longitude (longitude) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2\n", + " * time (time) object 8B 2000-01-16 12:00:00\n", + " height float64 8B ...\n", + "Data variables:\n", + " tas (time, latitude, longitude) float32 33kB dask.array<chunksize=(1, 64, 128), meta=np.ndarray>\n", + "Attributes: (12/54)\n", + " CCCma_model_hash: 3dedf95315d603326fde4f5340dc0519d80d10c0\n", + " CCCma_parent_runid: rc3-pictrl\n", + " CCCma_pycmor_hash: 33c30511acc319a98240633965a04ca99c26427e\n", + " CCCma_runid: rc3.1-his01\n", + " Conventions: CF-1.7 CMIP-6.2\n", + " YMDH_branch_time_in_child: 1850:01:01:00\n", + " ... ...\n", + " title: CanESM5 output prepared for CMIP6\n", + " tracking_id: hdl:21.14100/872062df-acae-499b-aa0f-9eaca76...\n", + " variable_id: tas\n", + " variant_label: r1i1p1f1\n", + " version: v20190429\n", + " status: 2019-10-25;created;by nhn2@columbia.edu
<xarray.Dataset> Size: 67kB\n", + "Dimensions: (time: 1, latitude: 64, longitude: 128)\n", + "Coordinates:\n", + " * time (time) object 8B 2000-01-16 12:00:00\n", + " * latitude (latitude) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86\n", + " * longitude (longitude) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2\n", + " height float64 8B ...\n", + "Data variables:\n", + " pr (time, latitude, longitude) float32 33kB dask.array<chunksize=(1, 64, 128), meta=np.ndarray>\n", + " tas (time, latitude, longitude) float32 33kB dask.array<chunksize=(1, 64, 128), meta=np.ndarray>\n", + "Attributes: (12/54)\n", + " CCCma_model_hash: 3dedf95315d603326fde4f5340dc0519d80d10c0\n", + " CCCma_parent_runid: rc3-pictrl\n", + " CCCma_pycmor_hash: 33c30511acc319a98240633965a04ca99c26427e\n", + " CCCma_runid: rc3.1-his01\n", + " Conventions: CF-1.7 CMIP-6.2\n", + " YMDH_branch_time_in_child: 1850:01:01:00\n", + " ... ...\n", + " title: CanESM5 output prepared for CMIP6\n", + " tracking_id: hdl:21.14100/872062df-acae-499b-aa0f-9eaca76...\n", + " variable_id: tas\n", + " variant_label: r1i1p1f1\n", + " version: v20190429\n", + " status: 2019-10-25;created;by nhn2@columbia.edu