Skip to content

Commit 89d1790

Browse files
lhoestqWauplinhanouticelinadavanstrienjulien-c
authored
[Dataset hub] more than storage: streaming, editing, connectors (#2052)
* index * wip * update libraries page * datasets streaming * datasets editing * Apply suggestions from code review Co-authored-by: Lucain <lucain@huggingface.co> Co-authored-by: célina <hanouticelina@gmail.com> Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com> Co-authored-by: Julien Chaumond <julien@huggingface.co> * Apply suggestions from code review Co-authored-by: Caleb Fahlgren <cfahlgren1@gmail.com> Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com> * try multiple columns for additional libraries features * HfFileSystem -> hffs * remove connector * ontent_defined_chunking -> use_content_defined_chunking * parquet visual * editing examples with huggingface_hub * typo * fix tips --------- Co-authored-by: Lucain <lucain@huggingface.co> Co-authored-by: célina <hanouticelina@gmail.com> Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com> Co-authored-by: Julien Chaumond <julien@huggingface.co> Co-authored-by: Caleb Fahlgren <cfahlgren1@gmail.com> Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com>
1 parent e4d6228 commit 89d1790

File tree

5 files changed

+368
-17
lines changed

5 files changed

+368
-17
lines changed

‎docs/hub/datasets-downloading.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Integrated libraries
44

5-
If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/Samsung/samsum?library=datasets) shows how to do so with 🤗 Datasets below.
5+
If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with `datasets` below.
66

77
<div class="flex justify-center">
88
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage.png"/>

‎docs/hub/datasets-editing.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Editing datasets
2+
3+
The [Hub](https://huggingface.co/datasets) enables collaborative curation of community and research datasets. We encourage you to explore the datasets available on the Hub and contribute to their improvement to help grow the ML community and accelerate progress for everyone. All contributions are welcome!
4+
5+
Start by [creating a Hugging Face Hub account](https://huggingface.co/join) if you don't have one yet.
6+
7+
## Edit using the Hub UI
8+
9+
> [!WARNING]
10+
> This feature is only available for CSV datasets for now.
11+
12+
The Hub's web interface allows users without any technical expertise to edit a dataset.
13+
14+
Open the dataset page and navigate to the **Data Studio** tab to begin editing.
15+
16+
<div class="flex justify-center">
17+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/data_studio_button-min.png"/>
18+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/data_studio_button_dark-min.png"/>
19+
</div>
20+
21+
Click on **Toggle edit mode** to enable dataset editing.
22+
23+
<div class="flex justify-center">
24+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/toggle_edit_button-min.png"/>
25+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/toggle_edit_button_dark-min.png"/>
26+
</div>
27+
28+
<div class="flex justify-center">
29+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/edit_cell_button-min.png"/>
30+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/edit_cell_button_dark-min.png"/>
31+
</div>
32+
33+
Edit as many cells as you want and finally click **Commit** to commit your changes and leave a commit message.
34+
35+
36+
<div class="flex justify-center">
37+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/commit_button-min.png"/>
38+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/commit_button_dark-min.png"/>
39+
</div>
40+
41+
<div class="flex justify-center">
42+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/commit_message-min.png"/>
43+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/commit_message_dark-min.png"/>
44+
</div>
45+
46+
## Using the `huggingface_hub` client library
47+
48+
The `huggingface_hub` library can manage Hub repositories including editing datasets.
49+
50+
For example here is how to edit a CSV file using the [Hugging Face FileSystem API](https://huggingface.co/docs/huggingface_hub/en/guides/hf_file_system):
51+
52+
```python
53+
from huggingface_hub import hffs
54+
55+
path = f"datasets/{repo_id}/data.csv"
56+
57+
with hffs.open(path, "r") as f:
58+
content = f.read()
59+
edited_content = content.replace("foo", "bar")
60+
with hffs.open(path, "w") as f:
61+
f.write(edited_content)
62+
```
63+
64+
You can also apply edit locally on your disk and commit the changes:
65+
66+
```python
67+
from huggingface_hub import hf_hub_download, upload_file
68+
69+
local_path = hf_hub_download(repo_id=repo_id, path_in_repo= "data.csv", repo_type="dataset")
70+
71+
with open(path, "r") as f:
72+
content = f.read()
73+
edited_content = content.replace("foo", "bar")
74+
with open(path, "w") as f:
75+
f.write(edited_content)
76+
77+
upload_file(repo_id=repo_id, path_in_repo=local_path, repo_type="dataset")
78+
```
79+
80+
> [!TIP]
81+
>
82+
> To have the entire dataset repository locally and edit many files at once, use `snapshot_download` and `upload_folder` instead of `hf_hub_download` and `upload_file`
83+
84+
85+
Visit [the client library's documentation](/docs/huggingface_hub/index) to learn more.
86+
87+
## Integrated libraries
88+
89+
If a dataset on the Hub is compatible with a [supported library](./datasets-libraries), loading, editing, and pushing the dataset takes just a few lines.
90+
91+
Here is how to edit a CSV file with Pandas:
92+
93+
```python
94+
import pandas as pd
95+
96+
# Load the dataset
97+
df = pd.read_csv(f"hf://datasets/{repo_id}/data.csv")
98+
99+
# Edit
100+
df = df.apply(...)
101+
102+
# Commit the changes
103+
df.to_csv(f"hf://datasets/{repo_id}/data.csv")
104+
```
105+
106+
Libraries like Polars and DuckDB also implement the `hf://` protocol to read, edit and write files on Hugging Face. And other libraries are useful to edit datasets made of many files like Spark, Dask or 🤗 Datasets. See the full list of supported libraries [here](./datasets-libraries)
107+
108+
For information on accessing the dataset on the website, you can click on the "Use this dataset" button on the dataset page to see how to do so.
109+
For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.
110+
111+
<div class="flex justify-center">
112+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage.png"/>
113+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-dark.png"/>
114+
</div>
115+
116+
<div class="flex justify-center">
117+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-modal.png"/>
118+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-modal-dark.png"/>
119+
</div>
120+
121+
## Only upload the new data
122+
123+
Hugging Face's storage is powered by [Xet](https://huggingface.co/docs/hub/en/xet), which uses chunk deduplication to make uploads more efficient.
124+
Unlike traditional cloud storage, Xet doesn't require the entire dataset to be re-uploaded to commit changes.
125+
Instead, it automatically detects which parts of the dataset have changed and instructs the client library only to upload the updated parts.
126+
To do that, Xet uses a smart algorithm to find chunks of 64kB that already exist on Hugging Face.
127+
128+
Let's see our previous example with Pandas:
129+
130+
```python
131+
import pandas as pd
132+
133+
# Load the dataset
134+
df = pd.read_csv(f"hf://datasets/{repo_id}/data.csv")
135+
136+
# Edit part of the dataset
137+
df = df.apply(...)
138+
139+
# Commit the changes
140+
df.to_csv(f"hf://datasets/{repo_id}/data.csv")
141+
```
142+
143+
This code first loads a dataset and then edits it.
144+
Once the edits are done, `to_csv()` materializes the file in memory, chunks it, asks Xet which chunks are already on Hugging Face and which chunks have changed, and then uploads only the new data.
145+
146+
## Optimized Parquet editing
147+
148+
The amount of data to upload depends on the edits and the file structure.
149+
150+
The Parquet format is columnar and compressed at the page level (pages are around ~1MB).
151+
We optimized Parquet for Xet with [Parquet Content Defined Chunking](https://huggingface.co/blog/parquet-cdc), which ensures unchanged data generally result in unchanged pages.
152+
153+
Check out if your library supports optimized Parquet in the [supported libraries](./datasets-libraries) page.
154+
155+
## Streaming
156+
157+
For big datasets, libraries with dataset streaming features for end-to-end streaming pipelines are recommended.
158+
In this case, the dataset processing runs progressively as the old data arrives and the new data is uploaded to the Hub.
159+
160+
Check out if your library supports streaming in the [supported libraries](./datasets-libraries) page.

‎docs/hub/datasets-libraries.md‎

Lines changed: 42 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,50 @@ The Datasets Hub has support for several libraries in the Open Source ecosystem.
44
Thanks to the [huggingface_hub Python library](/docs/huggingface_hub), it's easy to enable sharing your datasets on the Hub.
55
We're happy to welcome to the Hub a set of Open Source libraries that are pushing Machine Learning forward.
66

7+
## Libraries table
8+
79
The table below summarizes the supported libraries and their level of integration.
810

9-
| Library | Description | Download from Hub | Push to Hub |
10-
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ----------- |
11-
| [Argilla](./datasets-argilla) | Collaboration tool for AI engineers and domain experts that value high quality data. | ✅ | ✅ |
12-
| [Daft](./datasets-daft) | Data engine for large scale, multimodal data processing with a Python-native interface. | ✅ | ✅ |
13-
| [Dask](./datasets-dask) | Parallel and distributed computing library that scales the existing Python and PyData ecosystem. | ✅ | ✅ |
14-
| [Datasets](./datasets-usage) | 🤗 Datasets is a library for accessing and sharing datasets for Audio, Computer Vision, and Natural Language Processing (NLP). | ✅ | ✅ |
15-
| [Distilabel](./datasets-distilabel) | The framework for synthetic data generation and AI feedback. | ✅ | ✅ |
16-
| [DuckDB](./datasets-duckdb) | In-process SQL OLAP database management system. | ✅ | ✅ |
17-
| [Embedding Atlas](./datasets-embedding-atlas) | Interactive visualization and exploration tool for large embeddings. | ✅ | ❌ |
18-
| [fenic](./datasets-fenic) | PySpark-inspired DataFrame framework for building production AI and agentic applications. | ✅ | ❌ |
19-
| [FiftyOne](./datasets-fiftyone) | FiftyOne is a library for curation and visualization of image, video, and 3D data. | ✅ | ✅ |
20-
| [Pandas](./datasets-pandas) | Python data analysis toolkit. | ✅ | ✅ |
21-
| [Polars](./datasets-polars) | A DataFrame library on top of an OLAP query engine. | ✅ | ✅ |
22-
| [PyArrow](./datasets-pyarrow) | Apache Arrow is a columnar format and a toolbox for fast data interchange and in-memory analytics. | ✅ | ✅ |
23-
| [Spark](./datasets-spark) | Real-time, large-scale data processing tool in a distributed environment. | ✅ | ✅ |
24-
| [WebDataset](./datasets-webdataset) | Library to write I/O pipelines for large datasets. | ✅ | ❌ |
11+
| Library | Description | Download from Hub | Stream from Hub | Push to Hub | Stream to Hub | Optimized Parquet files |
12+
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------- | --------------- | ----------- | ------------- | ----------------------- |
13+
| [Argilla](./datasets-argilla) | Collaboration tool for AI engineers and domain experts that value high quality data. | ✅ | ❌ | ✅ | ❌ | ❌ |
14+
| [Daft](./datasets-daft) | Data engine for large scale, multimodal data processing with a Python-native interface. | ✅ | ✅ | ✅ | ✅ | ✅ |
15+
| [Dask](./datasets-dask) | Parallel and distributed computing library that scales the existing Python and PyData ecosystem. | ✅ | ✅ | ✅ | ✅ | ✅* |
16+
| [Datasets](./datasets-usage) | 🤗 Datasets is a library for accessing and sharing datasets for Audio, Computer Vision, and Natural Language Processing (NLP). | ✅ | ✅ | ✅ | ✅ | ✅ |
17+
| [Distilabel](./datasets-distilabel) | The framework for synthetic data generation and AI feedback. | ✅ | ❌ | ✅ | ❌ | ❌ |
18+
| [DuckDB](./datasets-duckdb) | In-process SQL OLAP database management system. | ✅ | ✅ | ❌ | ❌ | ❌ |
19+
| [Embedding Atlas](./datasets-embedding-atlas) | Interactive visualization and exploration tool for large embeddings. | ✅ | ✅ | ❌ | ❌ | ❌ |
20+
| [Fenic](./datasets-fenic) | PySpark-inspired DataFrame framework for building production AI and agentic applications. | ✅ | ✅ | ❌ | ❌ | ❌ |
21+
| [FiftyOne](./datasets-fiftyone) | FiftyOne is a library for curation and visualization of image, video, and 3D data. | ✅ | ✅ | ✅ | ❌ | ❌ |
22+
| [Pandas](./datasets-pandas) | Python data analysis toolkit. | ✅ | ❌ | ✅ | ❌ | ✅* |
23+
| [Polars](./datasets-polars) | A DataFrame library on top of an OLAP query engine. | ✅ | ✅ | ✅ | ❌ | ❌ |
24+
| [PyArrow](./datasets-pyarrow) | Apache Arrow is a columnar format and a toolbox for fast data interchange and in-memory analytics. | ✅ | ✅ | ✅ | ❌ | ✅* |
25+
| [Spark](./datasets-spark) | Real-time, large-scale data processing tool in a distributed environment. | ✅ | ✅ | ✅ | ✅ | ✅ |
26+
| [WebDataset](./datasets-webdataset) | Library to write I/O pipelines for large datasets. | ✅ | ✅ | ❌ | ❌ | ❌ |
27+
28+
_ * Requires passing extra arguments to write optimized Parquet files_
29+
30+
### Streaming
31+
32+
Dataset streaming allows iterating on a dataset from Hugging Face progressively without having to download it completely.
33+
It saves local disk space because the data is never on disk. It saves memory since only a small portion of the dataset is used at a time. And it saves time, since there is no need to download data before the CPU or GPU workload.
34+
35+
In addition to streaming *from* Hugging Face, many libraries also support streaming *back to* Hugging Face.
36+
Therefore, they can run end-to-end streaming pipelines: streaming from a source and writing to Hugging Face progressively, often overlapping the download, upload, and processing steps.
37+
38+
For more details on how to do streaming, check out the documentation of a library that support streaming (see table above) or the [streaming datasets](./datasets-streaming) documentation if you want to stream datasets from Hugging Face by yourself.
39+
40+
### Optimized Parquet files
41+
42+
Parquet files on Hugging Face are optimized to improve storage efficiency, accelerate downloads and uploads, and enable efficient dataset streaming and editing:
43+
44+
* [Parquet Content Defined Chunking](https://huggingface.co/blog/parquet-cdc) optimizes Parquet for [Xet](https://huggingface.co/docs/hub/en/xet/index), Hugging Face's storage backend. It accelerates uploads and downloads thanks to chunk-based deduplication and allows efficient file editing
45+
* Page index accelerates filters when streaming and enables efficient random access, e.g. in the [Dataset Viewer](https://huggingface.co/docs/dataset-viewer)
46+
47+
Some libraries require extra argument to write optimized Parquet files like `Pandas` and `PyArrow`:
48+
49+
* `use_content_defined_chunking=True` to enable Parquet Content Defined Chunking, for [deduplication](https://huggingface.co/blog/parquet-cdc) and [editing](./datasets-editing)
50+
* `write_page_index=True` to include a page index in the Parquet metadata, for [streaming and random access](./datasets-streaming)
2551

2652
## Integrating data libraries and tools with the Hub
2753

0 commit comments

Comments
 (0)