Skip to content

Commit ca2d986

Browse files
VeckoTheGeckoVeckoTheGecko
committed
Add data cataloging tutorial
1 parent 05544b4 commit ca2d986

1 file changed

Lines changed: 143 additions & 0 deletions

File tree

‎intermediate/cataloging.ipynb‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "0",
6+
"metadata": {
7+
"vscode": {
8+
"languageId": "plaintext"
9+
}
10+
},
11+
"source": [
12+
"# Data cataloguing for Xarray\n",
13+
"\n",
14+
"**Goals:** At the end of this tutorial, you'll have an overview of what data cataloging is, why it is done, what tools are available. You'll also know how to open and browse some data catalogues.\n",
15+
"\n",
16+
"> A data catalog is a detailed inventory of data assets within an organization. It helps users easily discover, understand, manage, curate and access data. -[IBM](https://www.ibm.com/think/topics/data-catalog)\n",
17+
"\n",
18+
"Within an organisation, you may work with a lot of different datasets. As someone managing this data, you might want the following things:\n",
19+
"\n",
20+
"- **An organized, easily browsable collection**: By easing discovery and loading of your datasets you can reduce the friction for those analysing the data. Users can eaily get an overview of available datasets, discover datasets of interest, and load them for analysis.\n",
21+
"- **Access control and logging**: You can limit who is able to access these datasets, and track when datasets are being accessed - keeping the data private, and providing metrics that can inform data management (e.g., by removing little-used datasets).\n",
22+
"- **Combine individual data assets**: Combine individual data assets that are similar into a larger object (e.g., combining individual time snapshots into a larger data cube)\n",
23+
"\n",
24+
"There are a bunch of solutions available in Xarray that meet some of these needs - we'll broadly call these \"data catalogs\". These solutions differ in functionality, and the correct data cataloguing solution depends on your needs as an institute.\n",
25+
"\n",
26+
"If you feel there are data cataloguing tools missing from this page, please submit a PR.\n",
27+
"\n",
28+
"## intake v2\n",
29+
"\n",
30+
"Intake is a lightweight Python package which allows one to specify data catalogues (or a hierarchy of catalogues) via YAML files. These YAML files describe:\n",
31+
"\n",
32+
"- how to load the dataset. This is done via \"readers\", which support different file formats, and support lazily defining additional transformations to the dataset using Dask\n",
33+
"- additional metadata for the dataset\n",
34+
"\n",
35+
"Uploading the catalogue file to a Git forge like GitHub allows for versioning of this file.\n",
36+
"\n",
37+
"This catalogue can be browsed in Python, or via third-party tools which parse the datalog, allowing the user to select and open a dataset. This tool has been adopted in the geosciences community for working with Xarray datasets, but is also flexible to work with other dataset types.\n",
38+
"\n",
39+
"There is no separate authentication layer in Intake, hence access control and logging depends entirely on the way the data is accessed. If the dataset is loaded from a file location on a mounted drive, then you need to have access to that mounted drive. If the dataset is located in an S3 bucket, you need to have the appropriate access credentials for the resource.\n",
40+
"\n",
41+
"Intake was originally developed within Anaconda by the team behind Dask.\n",
42+
"\n",
43+
"### Example using intake v2\n"
44+
]
45+
},
46+
{
47+
"cell_type": "code",
48+
"execution_count": null,
49+
"id": "526792bd",
50+
"metadata": {},
51+
"outputs": [],
52+
"source": []
53+
},
54+
{
55+
"cell_type": "markdown",
56+
"id": "d6767fe7",
57+
"metadata": {},
58+
"source": [
59+
"## STAC\n",
60+
"\n",
61+
"> SpatioTemporal Asset Catalogs\n",
62+
"> The STAC specification is a common language to describe geospatial information, so it can more easily be worked with, indexed, and discovered.\n",
63+
"> -stacspec.org\n",
64+
"\n",
65+
"Let's set the scene a bit: During sattelite data processing, many image files (predominantly TIFF) get generated. Each of these files can have different spatial and temporal extents, and over time these individual files can number in the millions, containing terrabytes of data in total. When doing analyses on these files, the first thing one needs to know is whether the file is even relevant to the analysis at hand (i.e., at least \"what is the spatial and temporal extent of the file?\"), however downloading the file just to check this metadata within is very wasteful. This is the context from which the STAC project was born.\n",
66+
"\n",
67+
"The goal of the STAC project is to make geospatial data more accesible by providing a specification to describe assets and their metadata. Adding this metadata alongside the asset (or lifting it out above the asset itself) allows:\n",
68+
"\n",
69+
"- The combining of assets into larger collections representing a larger dataset (e.g., a dataset with multiple time slices)\n",
70+
"- For more efficient workflows, as the whole asset doesn't need to be downloaded to check the metadata.\n",
71+
"\n",
72+
"\n",
73+
"Within the STAC project is:\n",
74+
"\n",
75+
"- **The STAC Specification**: A set of specifications describing (from most atomic to least) STAC Items, STAC Catalogs, and STAC collections. There is also a spec for the STAC API.\n",
76+
"- **Tooling for working with STAC**: More on this just now\n",
77+
"- **The community**\n",
78+
"\n",
79+
"There are various tools in the STAC ecosystem. [`odc-stac`](https://github.com/opendatacube/odc-stac), [`stacstac`](https://github.com/gjoseph92/stackstac) and [`xpystac`](https://github.com/stac-utils/xpystac) are all proojects that llow the opening of STAC items as Xarray datasets.\n",
80+
"\n",
81+
"### More resources\n",
82+
"\n",
83+
"- https://stacspec.org/en\n",
84+
"- https://stacindex.org/\n",
85+
"\n",
86+
"### Example using odc-stac\n"
87+
]
88+
},
89+
{
90+
"cell_type": "code",
91+
"execution_count": null,
92+
"id": "f6a76a8e",
93+
"metadata": {},
94+
"outputs": [],
95+
"source": []
96+
},
97+
{
98+
"cell_type": "markdown",
99+
"id": "2",
100+
"metadata": {},
101+
"source": [
102+
"## Commercial offering - Arraylake\n",
103+
"\n",
104+
"There are also commercial offerings available that have data cataloguing functionality.\n",
105+
"\n",
106+
"Arraylake is Earthmover's managed cloud platform built on top of the open-source Icechunk data format\n",
107+
"(where Icechunk is a project offering versioning of Zarr-like datasets).\n",
108+
"\n",
109+
"See [here](https://icechunk.io/en/latest/arraylake/) for a full comparison between Icechunk and Arraylake.\n",
110+
"\n",
111+
"A notable feature is that Arraylake integrates with [Earthmover's data marketplace](https://www.earthmover.io/marketplace/),\n",
112+
"powering data discovery and allowing for the dissemination of datasets (or subsets of datasets) to other interested users.\n",
113+
"\n",
114+
"\n",
115+
"\n",
116+
"## More resources\n",
117+
"\n",
118+
"https://guide.cloudnativegeo.org/cookbooks/zarr-stac-report/data-consumers/"
119+
]
120+
},
121+
{
122+
"cell_type": "markdown",
123+
"id": "06af9324",
124+
"metadata": {},
125+
"source": []
126+
}
127+
],
128+
"metadata": {
129+
"language_info": {
130+
"codemirror_mode": {
131+
"name": "ipython",
132+
"version": 3
133+
},
134+
"file_extension": ".py",
135+
"mimetype": "text/x-python",
136+
"name": "python",
137+
"nbconvert_exporter": "python",
138+
"pygments_lexer": "ipython3"
139+
}
140+
},
141+
"nbformat": 4,
142+
"nbformat_minor": 5
143+
}

0 commit comments

Comments
 (0)