feat: Add Apache TsFile format support #7933
Open
+440
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements support for Apache TsFile time-series data format (huggingface#7922). TsFile is a columnar storage format designed for IoT and time-series data, providing efficient compression and high query performance. Changes: - Add TsFileConfig and TsFile builder classes in packaged_modules/tsfile/ - Register tsfile module with .tsfile extension mapping - Add tsfile>=2.0.0 as optional dependency in setup.py - Add comprehensive test suite (11 tests) Features: - Automatic feature/schema inference from TsFile metadata - Support for table-model and tree-model TsFiles - Time-range filtering via start_time/end_time parameters - Column selection via columns parameter - Memory-efficient iterator-based reading with configurable batch_size Co-authored-by: TsFile Community
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Apache TsFile format support
Adds support for loading
.tsfiledatasets. Closes #7922.What's TsFile?
Apache TsFile is a columnar time-series format popular in IoT. The TsFile community requested this integration and offered to help maintain it.
What I did
Created a new
TsFilebuilder inpackaged_modules/tsfile/following the same pattern as HDF5. Registered the module and added.tsfileextension mapping. Also addedtsfile>=2.0.0as an optional dependency.The builder uses
tsfile.to_dataframe()with iterator mode for memory-efficient reading, then converts to PyArrow tables. Schema is inferred automatically from file metadata.Config options
batch_size- rows per batch (default 10000)table_name- which table to read (for multi-table files)columns- filter specific columnsstart_time/end_time- time-range filteringUsage
Tests
Added 11 tests covering config validation, basic loading, data integrity, feature inference, and error handling. All passing.