From 6336ead3321636f387985e47214b23628da66d04 Mon Sep 17 00:00:00 2001 From: ColinLee Date: Mon, 16 Nov 2026 10:47:29 +0800 Subject: [PATCH 1/3] docs: refresh README examples and encoding references Fix outdated Java API snippets (TSRecord, DataPoint, Path, writeRecord), the released dependency version, and the example class list. Align the C++ parallel-write API names and Chinese build instructions, and point the Python README at the real example and API modules. Update the CLI command table and encoding matrix to match current behavior and the current IoTDB reference documentation. --- README-zh.md | 37 +++++++++-------- README.md | 41 ++++++++++-------- cpp/README-zh.md | 30 +++++++++----- cpp/README.md | 6 +-- java/examples/readme.md | 89 +++++++++++++++++++--------------------- java/tsfile/README-zh.md | 36 ++++++++-------- java/tsfile/README.md | 22 +++++----- python/README-zh.md | 7 ++-- python/README.md | 2 +- 9 files changed, 144 insertions(+), 126 deletions(-) diff --git a/README-zh.md b/README-zh.md index 62fc18513f..29eaa5c904 100644 --- a/README-zh.md +++ b/README-zh.md @@ -92,7 +92,7 @@ TsFile 数据模型(Schema)定义了所有设备物理量的集合,如下 ### 编码和压缩 -TsFile 通过采用二阶差分编码、游程编码(RLE)、位压缩和 Snappy 等先进的编码和压缩技术,优化时序数据的存储和访问,并支持对时间戳列和数据值列进行单独编码,以实现更好的数据处理效能。 +TsFile 通过采用二阶差分编码(TS_2DIFF)、游程编码(RLE)、GORILLA、CHIMP、SPRINTZ、RLBE、字典编码(DICTIONARY)等先进的编码技术,以及 LZ4(默认)、Zstandard、Snappy、GZIP、LZMA2 等压缩算法,优化时序数据的存储和访问,并支持对时间戳列和数据值列进行单独编码,以实现更好的数据处理效能。 其独特之处在于编码算法专为时序数据特性设计,聚焦在时间属性和数据之间的相关性。 @@ -111,14 +111,18 @@ TsFile、CSV 和 Parquet 三种文件格式的比较 | 数据类型 | 推荐编码 | 推荐压缩算法 | |---------|------------|--------| +| BOOLEAN | RLE | LZ4 | | INT32 | TS_2DIFF | LZ4 | +| DATE | TS_2DIFF | LZ4 | | INT64 | TS_2DIFF | LZ4 | +| TIMESTAMP | TS_2DIFF | LZ4 | | FLOAT | GORILLA | LZ4 | | DOUBLE | GORILLA | LZ4 | -| BOOLEAN | RLE | LZ4 | -| TEXT | DICTIONARY | LZ4 | +| TEXT | PLAIN | LZ4 | +| STRING | PLAIN | LZ4 | +| BLOB | PLAIN | LZ4 | -更多类型的编码和压缩方式参见[文档](https://iotdb.apache.org/zh/UserGuide/latest/Basic-Concept/Encoding-and-Compression.html) +完整的编码与压缩支持矩阵参见[文档](https://iotdb.apache.org/zh/UserGuide/latest/Technical-Insider/Encoding-and-Compression.html) ## 开发和使用 TsFile @@ -131,27 +135,28 @@ TsFile、CSV 和 Parquet 三种文件格式的比较 ## 命令行工具(tsfile-cli) Apache TsFile 提供了命令行工具 `tsfile-cli`,这是一个单文件、对管道友好的工具,可直接在 -shell 中查看**并**导入 `.tsfile` 文件。读取类命令 -(`ls`、`meta`、`schema`、`stats`、`count`、`head`、`cat`、`sample`)将数据输出到 stdout、 -诊断信息输出到 stderr,因此可与 `awk`、`jq`、`sort` 等工具组合使用;`write` 命令则将 CSV -导入为新的 `.tsfile`。支持的输出格式:`csv`、`tsv`、`json`(NDJSON)、`table`。 +shell 中查看**并**导入 `.tsfile` 文件。`ls`、`meta`、`schema`、`stats`、`count`、`sketch`、 +`head`、`cat` 等命令将数据行输出到 stdout、诊断信息输出到 stderr,因此可与 `awk`、`jq`、 +`sort` 等工具组合使用;`export` 可将数据行或文件导出,`write` 命令则将 CSV 导入为新的 +`.tsfile`。支持的输出格式:`table`(默认)、`ndjson`、`csv`。 ### 命令 | 命令 | 作用 | |---|---| -| `ls` | 列出文件中的表(表模型)或设备(树模型),每行一个名称 | -| `meta` | 文件概览:数据模型、表 / 设备 / 序列数量、时间范围、文件大小 | +| `ls` | 以 `model, object` 行的形式列出文件中的表(表模型)或设备(树模型) | +| `meta` | 文件概览:文件大小、格式版本、数据模型 | | `schema` | 每条序列的数据类型、编码、压缩方式 | -| `stats` | 每条序列的统计信息:行数、时间范围、最小 / 最大值、首 / 末值、求和 | -| `count` | 每条序列的行数及总计 —— 直接读取统计信息,不扫描数据页 | +| `stats` | FIELD 统计信息:行数、空值数、时间范围、最小 / 最大值、首 / 末值、求和 | +| `count` | 对象 / 列计数——不生成汇总行 | +| `sketch` | 输出物理文件结构(chunk group、chunk、page),可用 `-o` 写入文件 | | `head` | 输出前 N 行(默认 10,可用 `-n` 调整) | -| `cat` | 流式输出所有匹配的行 | -| `sample` | 对行做可复现的蓄水池采样(`-n`、`--seed`) | +| `cat` | 输出所有匹配的行 | +| `export` | 用 `-o` 导出单个对象,或用 `--output-dir` 导出多个对象(`--type` 指定格式) | | `write` | 将 CSV 导入为新的表模型 `.tsfile` | -其中元数据类命令(`ls`、`meta`、`schema`、`stats`、`count`)无需解码数据页即可回答大部分问题, -而 `head`、`cat`、`sample` 则会读取真实的行数据。 +`ls`、`schema`、`meta` 用于只读元数据的发现;统计信息不可用时 `stats` 会扫描数据, +表模型的 `count` 会扫描行。`head`、`cat`、`export` 和 `sketch` 则读取实际数据或文件布局。 ### 示例 diff --git a/README.md b/README.md index e2fdeba2fa..a147dd0c48 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ TsFile adopts a columnar storage design, similar to other file formats, primaril ## Encoding and Compression -TsFile employs advanced encoding and compression techniques to optimize storage and access for time series data. It uses methods like run-length encoding (RLE), bit-packing, and Snappy for efficient compression, allowing separate encoding of timestamp and value columns for better data processing. Its unique encoding algorithms are designed specifically for the characteristics of time series data in IoT scenarios, focusing on regular time intervals and the correlation among series. +TsFile employs advanced encoding and compression techniques to optimize storage and access for time series data. Timestamps and value columns are encoded separately, using methods such as TS_2DIFF, run-length encoding (RLE), GORILLA, CHIMP, SPRINTZ, RLBE, and DICTIONARY, and compressed with LZ4 (the default), Zstandard, Snappy, GZIP, or LZMA2. Its unique encoding algorithms are designed specifically for the characteristics of time series data in IoT scenarios, focusing on regular time intervals and the correlation among series. Its uniqueness lies in the encoding algorithm designed specifically for time series data characteristics, focusing on the correlation between time attributes and data. @@ -111,14 +111,18 @@ Its development facilitates efficient data encoding, compression, and access, re | Data Type | Recommended Encoding | Recommended Compression | |---------|------------|--------| +| BOOLEAN | RLE | LZ4 | | INT32 | TS_2DIFF | LZ4 | +| DATE | TS_2DIFF | LZ4 | | INT64 | TS_2DIFF | LZ4 | +| TIMESTAMP | TS_2DIFF | LZ4 | | FLOAT | GORILLA | LZ4 | | DOUBLE | GORILLA | LZ4 | -| BOOLEAN | RLE | LZ4 | -| TEXT | DICTIONARY | LZ4 | +| TEXT | PLAIN | LZ4 | +| STRING | PLAIN | LZ4 | +| BLOB | PLAIN | LZ4 | -more see [Docs](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Encoding-and-Compression.html) +For the full encoding/compression matrix, see [Docs](https://iotdb.apache.org/UserGuide/latest/Technical-Insider/Encoding-and-Compression.html) ## Build and Use TsFile @@ -131,33 +135,36 @@ more see [Docs](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Encoding ## Command-Line Tool (tsfile-cli) Apache TsFile ships `tsfile-cli`, a single, pipe-friendly command-line tool for inspecting -**and** importing `.tsfile` files directly from the shell. Read commands (`ls`, `meta`, -`schema`, `stats`, `count`, `head`, `cat`, `sample`) print to stdout and diagnostics to -stderr, so they compose with `awk`, `jq`, `sort`, and friends; the `write` command imports -CSV into a new `.tsfile`. Output formats: `csv`, `tsv`, `json` (NDJSON), and `table`. +**and** importing `.tsfile` files directly from the shell. Commands such as `ls`, `meta`, +`schema`, `stats`, `count`, `sketch`, `head`, and `cat` print rows to stdout and diagnostics to +stderr, so they compose with `awk`, `jq`, `sort`, and friends; `export` writes rows or files, and +`write` imports CSV into a new `.tsfile`. Output formats: `table` (the default), `ndjson`, and +`csv`. ### Commands | Command | What it does | |---|---| -| `ls` | List the tables (table model) or devices (tree model), one name per line | -| `meta` | File summary: data model, table/device/series counts, time range, and file size | +| `ls` | List the tables (table model) or devices (tree model) as `model, object` rows | +| `meta` | File summary: size, format version, and data model | | `schema` | Per-series data type, encoding, and compression | -| `stats` | Per-series statistics: count, time range, min/max, first/last, and sum | -| `count` | Per-series row counts plus a total — read from statistics, without scanning pages | +| `stats` | FIELD statistics: count, null count, time range, min/max, first/last, and sum | +| `count` | Object/column counts — no synthetic total row | +| `sketch` | Print the physical file sketch (chunk groups, chunks, pages), optionally to `-o` | | `head` | Print the first N rows (default 10; `-n` to change) | -| `cat` | Stream every matching row | -| `sample` | Take a reproducible reservoir sample of rows (`-n`, `--seed`) | +| `cat` | Print every matching row | +| `export` | Export one object to `-o`, or multiple objects to `--output-dir`, using `--type` | | `write` | Import CSV into a new table-model `.tsfile` | -The metadata commands (`ls`, `meta`, `schema`, `stats`, `count`) answer most questions -without decoding data pages, while `head`, `cat`, and `sample` read the actual rows. +Use `ls`, `schema`, and `meta` for metadata-only discovery. `stats` can scan when +statistics are unavailable, and table-model `count` scans rows. `head`, `cat`, `export`, +and `sketch` read the actual data or layout. ### Examples ```bash tsfile-cli ls data.tsfile # list tables / devices -tsfile-cli meta data.tsfile # file overview (model, counts, time range, size) +tsfile-cli meta data.tsfile # file overview (size, version, model) tsfile-cli head -n 20 data.tsfile # first 20 rows tsfile-cli cat -m temp -m humidity -f csv data.tsfile # stream selected columns as CSV diff --git a/cpp/README-zh.md b/cpp/README-zh.md index e4c70bfc07..3ea8821976 100644 --- a/cpp/README-zh.md +++ b/cpp/README-zh.md @@ -43,7 +43,7 @@ TsFile_cpp 的性能基准测试位于 `./bench_mark` 目录。 我们使用 `clang-format` 来确保 C++ 代码遵循 `./clang-format` 文件中定义的一致规范(类似于 Google 风格)。 -`mvn spotless` 会使用 `clang-format v17.0.6` 来格式化 C++ 代码。在提交代码前,请先确认你环境里的 `clang-format` 版本与其一致,并且已经加入 `PATH`。 +`mvn spotless:apply` 会使用 `clang-format v17.0.6` 来格式化 C++ 代码。在提交代码前,请先确认你环境里的 `clang-format` 版本与其一致,并且已经加入 `PATH`。 `clang-format v17.0.6` 安装方式如下: @@ -75,7 +75,7 @@ mvn spotless:apply -P with-cpp 如果你需要临时跳过代码格式化检查,可以添加 `-Dspotless.skip=true`,例如: ```bash -mvn package -P with-cpp clean verify -Dspotless.skip=true +mvn clean verify -P with-cpp -Dspotless.skip=true ``` 欢迎提交任何 Bug 报告。 @@ -108,10 +108,10 @@ bash build.sh install 如果你安装了 Maven 工具,也可以运行: ```bash -mvn package -P with-cpp clean verify +mvn clean verify -P with-cpp ``` -构建完成后,可在 `./build` 目录下找到生成的共享库文件。 +`build.sh` 默认使用 Release,构建完成后可在 `./build/Release` 目录下找到生成的共享库文件。 在向 GitHub 提交代码之前,请确保 `mvn` 编译通过。 @@ -164,14 +164,23 @@ cmake .. -DENABLE_THREADS=OFF # 关闭——编译期剥离所有线程代码 ```cpp #include "common/global.h" -// 运行时开启或关闭并行写入(单核机器自动禁用) -storage::set_parallel_write_enabled(true); +// 运行时开启或关闭并行写入 / 读取路径。这两个开关在每次写入时以及查询 +// 打开迭代器时读取。 +common::set_parallel_write_enabled(true); +common::set_parallel_read_enabled(true); -// 设置工作线程数(必须在创建 TsFileWriter 之前调用) -storage::set_write_thread_count(4); +// 调整全局工作线程池(1..64)。库已初始化时立即生效;不得在 reader、 +// writer 或 result set 正在使用时调用。 +common::set_thread_count(4); ``` -默认情况下,当机器 CPU 核数大于 1 时自动启用并行写入,线程数设为硬件核数(上限 64)。 +C API 提供相同的开关:`tsfile_set_parallel_read_enabled()`、 +`tsfile_set_parallel_write_enabled()` 和 `tsfile_set_thread_count()`(以及对应的 +getter);Python 绑定提供 `set_parallel_read_enabled()`、 +`set_parallel_write_enabled()` 和 `set_thread_count()`。 + +并行读写默认开启,线程池为 6 个线程。使用 `ENABLE_THREADS=OFF` 时,setter +只记录取值,并行路径在编译期被移除。 ### 本地文件读取后端 @@ -210,7 +219,8 @@ reader 打开期间不得修改或截断文件。 bash build.sh ``` -即可在 `./examples/build` 目录下生成可执行文件。 +即可在 `./examples/build/Debug` 目录下生成可执行文件(`build.sh` 默认使用 Debug; +如需 Release,请修改脚本中的 `build_type`)。 ### 文件级 Properties diff --git a/cpp/README.md b/cpp/README.md index 09a80a8871..af97b8aabb 100644 --- a/cpp/README.md +++ b/cpp/README.md @@ -342,13 +342,13 @@ cmake .. -DENABLE_THREADS=OFF # disable — all thread code is stripped at comp // Enable or disable the parallel write / read paths at runtime. The flags are // consulted on every write and when a query opens its iterators. -storage::set_parallel_write_enabled(true); -storage::set_parallel_read_enabled(true); +common::set_parallel_write_enabled(true); +common::set_parallel_read_enabled(true); // Resize the global worker pool (1..64). Takes effect immediately when the // library has been initialized; do not call it while any reader, writer or // result set is in use. -storage::set_thread_count(4); +common::set_thread_count(4); ``` The C API exposes the same switches as diff --git a/java/examples/readme.md b/java/examples/readme.md index 6bd6597493..e485c66a8a 100644 --- a/java/examples/readme.md +++ b/java/examples/readme.md @@ -21,7 +21,7 @@ # Function ``` -The example is to show how to write and read a TsFile File. +The examples show how to write and read a TsFile file. ``` # Usage ## Dependencies with Maven @@ -31,54 +31,49 @@ The example is to show how to write and read a TsFile File. org.apache.tsfile tsfile - 2.1.0 + 2.5.0 ``` +## Write examples -## Run TsFileWrite.java - - - This class is to show how to write a TsFile. It provided two ways: - - The first one is using a JSON string for measurement(s). The JSON string is an array of JSON - objects(schema). The object must include the *measurement_id*, *datatype*, *encoding*, and - *compressor*. - - An example JSON string is provided in the comments in - the method - - public static void tsFileWriteWithJson() throws IOException,WriteProcessException - It uses this interface - - public void addMeasurementByJson(JSONObject measurement) throws WriteProcessException - An alternative way is to add these measurements directly(manually) by the second interface: - - public void addMeasurement(MeasurementSchema measurementSchema) throws WriteProcessException - - The method - - public static void tsFileWriteDirect() throws IOException,WriteProcessException - shows how to use that interface. - - Note that the measurements in the two methods are the same therefore there output TsFile should also be identical. - - - -## Run TsFileRead.java - - This class is to show how to read TsFile file named "testDirect.tsfile". - - The TsFile file "testDirect.tsfile" is generated from class TsFileWrite. - - It generates the same TsFile(testDirect.tsfile and testWithJson.tsfile) file by two different ways - - Run TsFileWrite to generate the testDirect.tsfile first - -## Run TsFileSequenceRead.java - - This class is to show the structure of a TsFile. - -### Notice -  For detail, please refer to https://github.com/apache/tsfile/blob/develop/README.md. +All writers below live in `src/main/java/org/apache/tsfile/` and can be run from +an IDE or with `java -cp org.apache.tsfile.`. + +| Class | Write model | API | Output file | +|---|---|---|---| +| `TsFileWriteWithTSRecord` | Non-aligned (tree model) | `TsFileWriter.registerTimeseries` + `TSRecord` | `Record.tsfile` | +| `TsFileWriteWithTablet` | Non-aligned (tree model) | `TsFileWriter.registerTimeseries` + `Tablet` | `Tablet.tsfile` | +| `TsFileWriteAlignedWithTSRecord` | Aligned | `TsFileWriter.registerAlignedTimeseries` + `TSRecord` | `alignedRecord.tsfile` | +| `TsFileWriteAlignedWithTablet` | Aligned and non-aligned in one file | `registerAlignedTimeseries` + `registerTimeseries` + `Tablet` | `alignedTablet.tsfile` | +| `TsFileForceAppendWrite` | Non-aligned (tree model) | `TsFileWriter.writeRecord` in batches | `test.tsfile` | + +`TSRecord` is the row-oriented API: one timestamp with a list of data points. +`Tablet` is the column-oriented batch API and is preferred for volume writes. +The aligned variants require every measurement of a device to share the same +timestamp column. + +## Read examples + +### Run TsFileRead.java + +`TsFileRead` queries `test.tsfile`, so run `TsFileForceAppendWrite` first to +generate it. The example shows a plain query, a time-range filter +(`GlobalTimeExpression` with `TimeFilterApi`), a value filter +(`SingleSeriesExpression` with `ValueFilterApi`), and a combined filter. + +### Run TsFileSequenceRead.java + +`TsFileSequenceRead` scans a TsFile from head to tail and prints its physical +layout: magic strings, metadata positions, chunk groups, chunks, and pages. Pass +the file to scan as the first argument; it defaults to `test.tsfile`. + +## V4 examples + +The `v4/` package contains examples for the newer reader/writer APIs: +`ITsFileReaderAndITsFileWriter`, `WriteTabletWithITsFileWriter`, +`TsFileTreeReaderExample`, and `TagFilterExample`. + +### Notice + For detail, please refer to https://github.com/apache/tsfile/blob/develop/README.md. diff --git a/java/tsfile/README-zh.md b/java/tsfile/README-zh.md index c470828a8e..157f5ac9e3 100644 --- a/java/tsfile/README-zh.md +++ b/java/tsfile/README-zh.md @@ -31,16 +31,16 @@ ### 在 Maven 中添加 TsFile 依赖 -当前发布版本是 `2.1.0`,可以这样引用 +当前发布版本是 `2.5.0`,可以这样引用 ```xml org.apache.tsfile tsfile - 2.1.0 + 2.5.0 - + ``` @@ -60,25 +60,25 @@ 2. 注册时间序列 ```java - List schema1 = new ArrayList<>(); - schemas.add(new MeasurementSchema("电压", TSDataType.FLOAT)); - schemas.add(new MeasurementSchema("电流", TSDataType.FLOAT)); + List schema1 = new ArrayList<>(); + schema1.add(new MeasurementSchema("电压", TSDataType.FLOAT)); + schema1.add(new MeasurementSchema("电流", TSDataType.FLOAT)); tsFileWriter.registerTimeseries(new Path("太阳能板1"), schema1); - List schema2 = new ArrayList<>(); - schemas.add(new MeasurementSchema("电压", TSDataType.FLOAT)); - schemas.add(new MeasurementSchema("电流", TSDataType.FLOAT)); - schemas.add(new MeasurementSchema("风速", TSDataType.FLOAT)); + List schema2 = new ArrayList<>(); + schema2.add(new MeasurementSchema("电压", TSDataType.FLOAT)); + schema2.add(new MeasurementSchema("电流", TSDataType.FLOAT)); + schema2.add(new MeasurementSchema("风速", TSDataType.FLOAT)); tsFileWriter.registerTimeseries(new Path("风机1"), schema2); ``` 3. 写入数据 ```java - TSRecord tsRecord = new TSRecord(1, "太阳能板1"); - tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "电压", 1.1f)); - tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "电流", 2.2f)); - tsFileWriter.write(tsRecord); + TSRecord tsRecord = new TSRecord("太阳能板1", 1); + tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "电压", "1.1")); + tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "电流", "2.2")); + tsFileWriter.writeRecord(tsRecord); ``` 4. 调用`close`方法来关闭文件,关闭后才能进行查询。 @@ -102,15 +102,15 @@ ```java TsFileSequenceReader reader = new TsFileSequenceReader(path); - TsFileReader tsFileReader = new TsFileReader(reader); + TsFileReader tsFileReader = new TsFileReader(reader); ``` 2. 构造查询条件 ```java ArrayList paths = new ArrayList<>(); - paths.add(new Path("太阳能板1", "电压")); - paths.add(new Path("太阳能板1", "电流")); + paths.add(new Path("太阳能板1", "电压", true)); + paths.add(new Path("太阳能板1", "电流", true)); IExpression timeFilter = BinaryExpression.and( new GlobalTimeExpression(TimeFilterApi.gtEq(1L)), @@ -122,7 +122,7 @@ 3. 查询数据 ```java - QueryDataSet queryDataSet = readTsFile.query(queryExpression); + QueryDataSet queryDataSet = tsFileReader.query(queryExpression); while (queryDataSet.hasNext()) { queryDataSet.next(); } diff --git a/java/tsfile/README.md b/java/tsfile/README.md index ec896ca279..750b080cc3 100644 --- a/java/tsfile/README.md +++ b/java/tsfile/README.md @@ -31,16 +31,16 @@ ### Add TsFile as a dependency in Maven -The current version is `2.1.0` +The current version is `2.5.0` ```xml org.apache.tsfile tsfile - 2.1.0 + 2.5.0 - + ``` ### TsFile Java API @@ -75,10 +75,10 @@ Data written is through TsFileWriter. 3. Write data ```java - TSRecord tsRecord = new TSRecord(1, "solarpanel1"); - tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "voltage", 1.1f)); - tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "electricity", 2.2f)); - tsFileWriter.write(tsRecord); + TSRecord tsRecord = new TSRecord("solarpanel1", 1); + tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "voltage", "1.1")); + tsRecord.addTuple(DataPoint.getDataPoint(TSDataType.FLOAT, "electricity", "2.2")); + tsFileWriter.writeRecord(tsRecord); ``` 4. Close TsFileWriter, only closed TsFile could be queried. @@ -102,15 +102,15 @@ Data query is through TsFileReader. ```java TsFileSequenceReader reader = new TsFileSequenceReader(path); - TsFileReader tsFileReader = new TsFileReader(reader); + TsFileReader tsFileReader = new TsFileReader(reader); ``` 2. Construct query expression, including predicate and filter ```java ArrayList paths = new ArrayList<>(); - paths.add(new Path("solarpanel1", "voltage")); - paths.add(new Path("solarpanel1", "electricity")); + paths.add(new Path("solarpanel1", "voltage", true)); + paths.add(new Path("solarpanel1", "electricity", true)); IExpression timeFilter = BinaryExpression.and( new GlobalTimeExpression(TimeFilterApi.gtEq(1L)), @@ -122,7 +122,7 @@ Data query is through TsFileReader. 3. Query data ```java - QueryDataSet queryDataSet = readTsFile.query(queryExpression); + QueryDataSet queryDataSet = tsFileReader.query(queryExpression); while (queryDataSet.hasNext()) { queryDataSet.next(); } diff --git a/python/README-zh.md b/python/README-zh.md index 59d2eb0f1a..c95bccbfb6 100644 --- a/python/README-zh.md +++ b/python/README-zh.md @@ -32,10 +32,11 @@ 本目录包含 TsFile 的 Python 实现版本。Python 版本基于 C++ 版本构建,并通过 Cython 包将 TsFile 的读写能力集成到 Python 环境中。用户可以像在 Pandas 中使用 read_csv 和 write_csv 一样,方便地读取和写入 TsFile。 源代码位于 `./tsfile` 目录。 -以 `.pyx` 和 `.pyd` 结尾的文件为使用 Cython 编写的封装代码。 -`tsfile/tsfile.py` 中定义了一些对用户开放的接口。 +以 `.pyx` 结尾的文件为使用 Cython 编写的封装代码;`tsfile/__init__.py` 导出的公共 API +以及纯 Python 模块(`constants.py`、`schema.py`、`tablet.py`、`tsfile_table_writer.py` +和 `dataset/` 包)定义了面向用户的接口。 -你可以在 `./examples/examples.py` 中找到读写示例。 +你可以在 `examples/example.py` 中找到读写示例。 --- diff --git a/python/README.md b/python/README.md index 7980e91db2..fcc46ea8db 100644 --- a/python/README.md +++ b/python/README.md @@ -32,7 +32,7 @@ This directory contains the Python implementation of TsFile. The Python version is built on the CPP version and uses the Cython package to integrate TsFile's read and write capabilities into the Python environment. Users can read and write TsFile as easily as they use read_csv and write_csv in Pandas. -The source code can be found in the `./tsfile` directory. Files ending with `.pyx` and `.pyd` are wrapper code written in Cython. The `tsfile/tsfile.py` defines some user interfaces. You can find some examples of reading and writing in the `.examples/examples.py`. +The source code can be found in the `./tsfile` directory. Files ending with `.pyx` are Cython wrapper code, and `tsfile/__init__.py` exports the public API together with the pure-Python modules (`constants.py`, `schema.py`, `tablet.py`, `tsfile_table_writer.py`, and the `dataset/` package). You can find an example of reading and writing in `examples/example.py`. ## How to make contributions From 3dcf62591cbe269247670b9c2b3d8804fb9f2f9f Mon Sep 17 00:00:00 2001 From: ColinLee Date: Mon, 16 Nov 2026 10:47:33 +0800 Subject: [PATCH 2/3] ci: trigger workflows from changed language sources Replace broad paths-ignore filters with per-language path whitelists so documentation-only changes and unrelated language directories do not start builds. Keep shared build inputs (pom.xml, mvnw, .mvn, checkstyle.xml, and the compatibility matrix) in the workflows that consume them, and document the maintenance rule in CLAUDE.md. Build the native library without C++ test targets in the Go and CodeQL workflows, which do not run C++ unit tests. --- .github/workflows/code-coverage.yml | 30 ++++++++++++++++++--- .github/workflows/codeql.yml | 28 ++++++++++++++++--- .github/workflows/compatibility-test.yml | 2 ++ .github/workflows/cpp-dependency-source.yml | 4 +++ .github/workflows/cppcheck.yml | 24 +++++++++++++++-- .github/workflows/unit-test-cpp-msvc.yml | 26 +++++++++++++----- .github/workflows/unit-test-cpp.yml | 26 +++++++++++++----- .github/workflows/unit-test-go.yml | 8 +++--- .github/workflows/unit-test-java.yml | 28 +++++++++++++------ .github/workflows/unit-test-python-msvc.yml | 26 +++++++++++++----- .github/workflows/unit-test-python.yml | 26 +++++++++++++----- CLAUDE.md | 4 +++ 12 files changed, 188 insertions(+), 44 deletions(-) diff --git a/.github/workflows/code-coverage.yml b/.github/workflows/code-coverage.yml index 7361cf2848..be9855d495 100644 --- a/.github/workflows/code-coverage.yml +++ b/.github/workflows/code-coverage.yml @@ -4,14 +4,36 @@ on: branches: - develop - rc/* - paths-ignore: - - 'docs/**' + paths: + - '.github/workflows/code-coverage.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'checkstyle.xml' + - 'codecov.yml' + - 'compat/compatibility-matrix.csv' + - 'java/**' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' pull_request: branches: - develop - rc/* - paths-ignore: - - 'docs/**' + paths: + - '.github/workflows/code-coverage.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'checkstyle.xml' + - 'codecov.yml' + - 'compat/compatibility-matrix.csv' + - 'java/**' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' jobs: codecov: diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 46b7d205aa..6ff6e6d548 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -16,12 +16,34 @@ on: branches: - develop - rc/* + paths: + - '.github/workflows/codeql.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'java/**' + - 'cpp/**' + - 'go/**' + - 'python/**' + - '!cpp/docs/**' + - '!**.md' pull_request: branches: - develop - rc/* - paths-ignore: - - 'docs/**' + paths: + - '.github/workflows/codeql.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'java/**' + - 'cpp/**' + - 'go/**' + - 'python/**' + - '!cpp/docs/**' + - '!**.md' jobs: analyze: @@ -89,7 +111,7 @@ jobs: # The Go bridge needs headers and libtsfile under cpp/target/build. - name: Build native library for Go if: matrix.language == 'go' - run: ./mvnw -P with-cpp -DskipTests -Dspotless.skip=true package + run: ./mvnw -P with-cpp -Dbuild.test=OFF -DskipTests -Dspotless.skip=true package # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL diff --git a/.github/workflows/compatibility-test.yml b/.github/workflows/compatibility-test.yml index 3688a3fd03..8ed1325ad2 100644 --- a/.github/workflows/compatibility-test.yml +++ b/.github/workflows/compatibility-test.yml @@ -45,6 +45,7 @@ on: - 'cpp/cmake/**' - 'cpp/third_party/**' - 'cpp/test/reader/table_view/encoding_compression_compatibility_test.cc' + - '!**.md' pull_request: branches: - develop @@ -73,6 +74,7 @@ on: - 'cpp/cmake/**' - 'cpp/third_party/**' - 'cpp/test/reader/table_view/encoding_compression_compatibility_test.cc' + - '!**.md' workflow_dispatch: inputs: forceUpdates: diff --git a/.github/workflows/cpp-dependency-source.yml b/.github/workflows/cpp-dependency-source.yml index 80fa184d05..b3facf1374 100644 --- a/.github/workflows/cpp-dependency-source.yml +++ b/.github/workflows/cpp-dependency-source.yml @@ -26,6 +26,8 @@ on: paths: - '.github/workflows/cpp-dependency-source.yml' - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' - 'packaging/tests/test_native_install.py' - 'pom.xml' pull_request: @@ -37,6 +39,8 @@ on: paths: - '.github/workflows/cpp-dependency-source.yml' - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' - 'packaging/tests/test_native_install.py' - 'pom.xml' workflow_dispatch: diff --git a/.github/workflows/cppcheck.yml b/.github/workflows/cppcheck.yml index 5ce962181e..53453d6d35 100644 --- a/.github/workflows/cppcheck.yml +++ b/.github/workflows/cppcheck.yml @@ -26,9 +26,19 @@ on: paths: - '.github/workflows/cppcheck.yml' - '.mvn/**' - - 'cpp/**' - 'mvnw' + - 'mvnw.cmd' - 'pom.xml' + - 'cpp/CMakeLists.txt' + - 'cpp/cmake/**' + - 'cpp/src/**' + - 'cpp/tools/**' + - 'cpp/examples/**' + - 'cpp/third_party/**' + - 'cpp/build.sh' + - 'cpp/pom.xml' + - '!cpp/tools/skills/**' + - '!**.md' pull_request: branches: - develop @@ -38,9 +48,19 @@ on: paths: - '.github/workflows/cppcheck.yml' - '.mvn/**' - - 'cpp/**' - 'mvnw' + - 'mvnw.cmd' - 'pom.xml' + - 'cpp/CMakeLists.txt' + - 'cpp/cmake/**' + - 'cpp/src/**' + - 'cpp/tools/**' + - 'cpp/examples/**' + - 'cpp/third_party/**' + - 'cpp/build.sh' + - 'cpp/pom.xml' + - '!cpp/tools/skills/**' + - '!**.md' workflow_dispatch: concurrency: diff --git a/.github/workflows/unit-test-cpp-msvc.yml b/.github/workflows/unit-test-cpp-msvc.yml index b30e12748b..04526d2d40 100644 --- a/.github/workflows/unit-test-cpp-msvc.yml +++ b/.github/workflows/unit-test-cpp-msvc.yml @@ -10,18 +10,32 @@ on: - develop - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-cpp-msvc.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'compat/compatibility-matrix.csv' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' pull_request: branches: - develop - dev/* - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-cpp-msvc.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'compat/compatibility-matrix.csv' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' # Enable manually starting builds. workflow_dispatch: diff --git a/.github/workflows/unit-test-cpp.yml b/.github/workflows/unit-test-cpp.yml index 6b893954c8..3df95cb972 100644 --- a/.github/workflows/unit-test-cpp.yml +++ b/.github/workflows/unit-test-cpp.yml @@ -9,18 +9,32 @@ on: - develop - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-cpp.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'compat/compatibility-matrix.csv' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' pull_request: branches: - develop - dev/* - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-cpp.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'compat/compatibility-matrix.csv' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' # Enable manually starting builds, and allow forcing updating of SNAPSHOT dependencies. workflow_dispatch: inputs: diff --git a/.github/workflows/unit-test-go.yml b/.github/workflows/unit-test-go.yml index 53dae64014..0d5c10be0f 100644 --- a/.github/workflows/unit-test-go.yml +++ b/.github/workflows/unit-test-go.yml @@ -23,11 +23,13 @@ on: - "go/**" - "cpp/src/cwrapper/**" - ".github/workflows/unit-test-go.yml" + - "!**.md" push: branches: [develop] paths: - "go/**" - "cpp/src/cwrapper/**" + - "!**.md" jobs: test: @@ -83,12 +85,12 @@ jobs: run: | if [[ "${{ runner.os }}" == "Windows" ]]; then if [[ "${{ matrix.toolchain }}" == "msvc" ]]; then - ./mvnw.cmd -P with-cpp -Dcpp.toolchain=msvc -Dbuild.type=Release -DskipTests -Dspotless.skip=true package + ./mvnw.cmd -P with-cpp -Dcpp.toolchain=msvc -Dbuild.type=Release -Dbuild.test=OFF -DskipTests -Dspotless.skip=true package else - ./mvnw.cmd -P with-cpp -Dcpp.toolchain=mingw -Dbuild.type=Release -DskipTests -Dspotless.skip=true package + ./mvnw.cmd -P with-cpp -Dcpp.toolchain=mingw -Dbuild.type=Release -Dbuild.test=OFF -DskipTests -Dspotless.skip=true package fi else - ./mvnw -P with-cpp -DskipTests -Dspotless.skip=true package + ./mvnw -P with-cpp -Dbuild.test=OFF -DskipTests -Dspotless.skip=true package fi - name: Expose Windows native libraries if: matrix.toolchain == 'mingw' diff --git a/.github/workflows/unit-test-java.yml b/.github/workflows/unit-test-java.yml index b84954ec00..90dea6ca70 100644 --- a/.github/workflows/unit-test-java.yml +++ b/.github/workflows/unit-test-java.yml @@ -9,19 +9,31 @@ on: - develop - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'cpp/**' - - 'python/**' + paths: + - '.github/workflows/unit-test-java.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'checkstyle.xml' + - 'compat/compatibility-matrix.csv' + - 'java/**' + - '!**.md' pull_request: branches: - develop - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'cpp/**' - - 'python/**' + paths: + - '.github/workflows/unit-test-java.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'checkstyle.xml' + - 'compat/compatibility-matrix.csv' + - 'java/**' + - '!**.md' # Enable manually starting builds, and allow forcing updating of SNAPSHOT dependencies. workflow_dispatch: inputs: diff --git a/.github/workflows/unit-test-python-msvc.yml b/.github/workflows/unit-test-python-msvc.yml index 80e3b61f7b..4f521fad13 100644 --- a/.github/workflows/unit-test-python-msvc.yml +++ b/.github/workflows/unit-test-python-msvc.yml @@ -15,18 +15,32 @@ on: - develop - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-python-msvc.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'python/**' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' pull_request: branches: - develop - dev/* - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-python-msvc.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'python/**' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' # Enable manually starting builds. workflow_dispatch: diff --git a/.github/workflows/unit-test-python.yml b/.github/workflows/unit-test-python.yml index 66eca86997..c7ad7528f6 100644 --- a/.github/workflows/unit-test-python.yml +++ b/.github/workflows/unit-test-python.yml @@ -9,18 +9,32 @@ on: - develop - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-python.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'python/**' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' pull_request: branches: - develop - dev/* - iotdb - rc/* - paths-ignore: - - 'docs/**' - - 'java/**' + paths: + - '.github/workflows/unit-test-python.yml' + - '.mvn/**' + - 'mvnw' + - 'mvnw.cmd' + - 'pom.xml' + - 'python/**' + - 'cpp/**' + - '!cpp/docs/**' + - '!**.md' # Enable manually starting builds, and allow forcing updating of SNAPSHOT dependencies. workflow_dispatch: inputs: diff --git a/CLAUDE.md b/CLAUDE.md index 9a157d97c9..9ef910ee4b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,6 +95,10 @@ For changes requiring full verification, run: - Inspect the applicable workflows in `.github/workflows/` and verify their additional checks, including Python binding ASan/LSan coverage as documented in `python/SANITIZER.md`, compatibility tests, and static analysis. A successful ordinary Maven run does not establish that sanitizer checks passed; the Python sanitizer build also requires its separate instrumentation check and ownership test run. - Check the actual test reports and exit statuses for Maven, ASan, formatting, and other required checks. Record the commands and results before pushing. If a required check fails, is skipped, or cannot run in the current environment, report the blocker and do not push until it has passed on a supported environment. +## CI Path Filters + +The GitHub Actions workflows under `.github/workflows/` use per-language `paths` whitelists so that documentation-only changes and unrelated language directories do not trigger builds. When adding a directory or shared build input that a workflow consumes, update that workflow's `paths` list in the same change. Markdown (`**.md`), `docs/**`, `skills/**`, `cpp/docs/**`, and other languages' source directories are intentionally excluded from the language test workflows. + ## Internationalization Java logs and exception messages use a runtime `ResourceBundle` approach so the same JAR can emit English or Simplified Chinese based on a JVM startup property. See `java/CLAUDE.md` for the convention. Switch language with `-Dtsfile.locale=zh`. From 7c390e7d2ed36b915b9e5eef0eecdf53a459ff64 Mon Sep 17 00:00:00 2001 From: ColinLee Date: Mon, 16 Nov 2026 11:21:26 +0800 Subject: [PATCH 3/3] ci: include packaging sources in CodeQL path filters Packaging scripts and tests are Python sources analyzed by CodeQL. Add packaging/** to the push and pull-request allowlists while retaining the Markdown exclusion. --- .github/workflows/codeql.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 6ff6e6d548..cb96c06583 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -26,6 +26,7 @@ on: - 'cpp/**' - 'go/**' - 'python/**' + - 'packaging/**' - '!cpp/docs/**' - '!**.md' pull_request: @@ -42,6 +43,7 @@ on: - 'cpp/**' - 'go/**' - 'python/**' + - 'packaging/**' - '!cpp/docs/**' - '!**.md'