diff --git a/cpp/tools/README.md b/cpp/tools/README.md index 6c4590fce..ab09adc78 100644 --- a/cpp/tools/README.md +++ b/cpp/tools/README.md @@ -96,11 +96,11 @@ Exit codes: `0` success, `1` usage/argument error, `2` file open/corrupt, | `count` | Object/column counts; no synthetic summary row | | `sketch` | Print the physical file sketch, optionally to `-o` | | `head` | First N rows (default 10; use `-n`) | -| `cat` | All matching rows, streamed (`table` format buffers to align columns) | +| `cat` | All matching rows, buffered before publishing | | `export` | Export one object to `-o`, or multiple objects to `--output-dir`, using `--type` | -The metadata commands (`ls` / `schema` / `meta` / `stats` / `count`) answer most questions -without decoding data pages. +Use `ls`, `schema`, and `meta` for metadata-only discovery. `stats` can scan +when statistics are unavailable, and table-model `count` scans rows. Shared options: @@ -114,14 +114,17 @@ Shared options: | `--tag-filter C OP [V]` | Table TAG predicate for row reads; `OP` is `eq`, `neq`, `regexp`, `is-null`, or `not-null` | | `--tag-match all\|any` | Required when more than one `--tag-filter` is supplied | -`ndjson` output emits one JSON object per line; numbers/booleans are bare, other values are -quoted, nulls are `null`, and non-finite floats become `null`. CSV output follows RFC 4180. +`ndjson` output emits one JSON object per line; INT64/TIMESTAMP values are decimal +strings, BOOLEAN/INT32/FLOAT/DOUBLE are bare values, nulls are `null`, and +non-finite floats become `null`. CSV output follows RFC 4180. CSV nulls are unquoted `\N`; empty strings are quoted as `""`. Non-null STRING/TEXT values beginning with `\` gain one extra leading `\`, using the same encoding rule -as CSV input described below. Timestamps are raw int64 values. The `table` format -uses a temporary spool to align columns with bounded memory; prefer `csv`/`ndjson` -when temporary disk use is undesirable. `sketch` -does not accept `--format`. +as CSV input described below. Timestamps are raw int64 values. Row commands +currently stage the complete formatted result in memory before publishing it, +including CSV/NDJSON and `export`. The `table` formatter additionally uses a +temporary spool to align columns; this does not bound memory for the complete +output path. Use row, time, and TAG bounds for large reads. `sketch` does not +accept `--format`. Text output preserves well-formed UTF-8 and replaces malformed UTF-8 in names and text with U+FFFD (`�`). This also applies to sketch output and export @@ -165,6 +168,11 @@ By default each column uses the engine's default encoding and compression for it `--encoding` and `--compression` override by canonical data type, applying to every declared TAG/FIELD of that type; they do not target individual columns and do not affect `time`. +The CLI currently rejects explicit ALP, GORILLA_V1, and BOOLEAN/FLOAT/DOUBLE RLE +overrides even though the SDK supports them. ALP files can be read, but `schema` +currently displays their encoding as `UNKNOWN`. Native build options also +control compression availability; LZMA2 is opt-in. + | Option | Meaning | |---|---| | `--table ` | Output table name (lower-cased) | @@ -226,18 +234,17 @@ the skill just means placing it there — either project-level or user-level: ```bash # project-level (this repository only) -mkdir -p .claude/skills/tsfile-cli -cp cpp/tools/skills/tsfile-cli/SKILL.md .claude/skills/tsfile-cli/SKILL.md +mkdir -p .claude/skills +cp -R cpp/tools/skills/tsfile-cli .claude/skills/ # or user-level (available in all your projects) -mkdir -p ~/.claude/skills/tsfile-cli -cp cpp/tools/skills/tsfile-cli/SKILL.md ~/.claude/skills/tsfile-cli/SKILL.md +mkdir -p ~/.claude/skills +cp -R cpp/tools/skills/tsfile-cli ~/.claude/skills/ ``` -> The installed `SKILL.md` must begin with its YAML front-matter (`--- … ---`) for the -> assistant to detect it. The in-repo copy carries an Apache license header comment above -> the front-matter; if discovery fails, delete that leading `` block from the -> installed copy so `---` is the first line. +Copy the entire directory: `SKILL.md` loads the bundled `references/` files. +The current entrypoint already begins with YAML front-matter and keeps its +Apache license as YAML comments; no header removal is needed. Start a new assistant session afterward. The skill then activates automatically when you ask to inspect or import a `.tsfile`; you can also invoke it explicitly (e.g. "use the diff --git a/cpp/tools/skills/tsfile-cli/SKILL.md b/cpp/tools/skills/tsfile-cli/SKILL.md index a52aabbbb..f81013f4b 100644 --- a/cpp/tools/skills/tsfile-cli/SKILL.md +++ b/cpp/tools/skills/tsfile-cli/SKILL.md @@ -18,7 +18,7 @@ name: tsfile-cli description: >- Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect, - preview, export, or create an Apache TsFile; report metadata or per-series + preview, export, or create an Apache TsFile; report metadata or per-column counts; or use its explicit single-table CSV write command. --- @@ -41,14 +41,17 @@ Use this skill only for the C++ `tsfile-cli` binary. Route Java checks or backfill to the top-level `tsfile` skill when it is installed. The names overlap but the semantics do not: `tsfile-cli count` is a read-only -per-series report, not the Java table point-count property tool. `tsfile-cli +per-column report, not the Java table point-count property tool. `tsfile-cli write` is the C++ binary's narrow one-file/stream, one-table CSV import; it does not replace the Java batch and format-aware import tools. ## Binary -- Name `tsfile-cli` (CMake target `tsfile_cli`). Find: `ls cpp/build/*/bin/tsfile-cli`. +- Name `tsfile-cli` (CMake target `tsfile_cli`). Check `PATH`, + `cpp/build/*/bin/tsfile-cli`, or `cpp/target/build/bin/tsfile-cli`. - Build only if missing: `cd cpp && bash build.sh -t=Debug`. +- Use `tsfile-cli --help` for the installed binary's supported + syntax and result fields before choosing flags. ## Read @@ -62,25 +65,27 @@ does not replace the Java batch and format-aware import tools. | `stats` | FIELD statistics with null counts and source | maybe | | `count` | object/column counts; no summary row | maybe | | `head` | first N rows (default 10, `-n`) | yes | -| `cat` | all matching rows (streamed; `table` format buffers) | yes | +| `cat` | all matching rows (buffered) | yes | | `sketch` | physical layout text from the bound `printSketch` behavior | no FIELD decode | | `export` | writes one object file or a numbered multi-object directory | yes | -Prefer metadata verbs (`ls/schema/meta/stats/count`) before row verbs so the -file model, scope, and columns are known. +Inspect `ls`, `schema`, and `meta` before querying rows to identify the model, +objects, and columns. `stats` can scan when statistics are unavailable, and +table-model `count` scans rows; neither is guaranteed to be metadata-only. For `head` and `cat`, omit `-d/-t` only when the selected model has exactly one -accessible object. Multi-object files require explicit scope. `export` always +accessible object. Multi-object files require explicit scope. `schema`, +`stats`, and `count` visit every object when scope is omitted. `export` always requires explicit scope. ``` -opts: -f table|ndjson|csv (default table) +opts: -f table|ndjson|csv (default table, including pipes) -d | -t (mutually exclusive) -m repeat for projection or metadata filtering; no comma lists -n N · --offset N · --start · --end for head/cat/export --tag-filter C OP [V] where OP=eq|neq|regexp|is-null|not-null --tag-match all|any when two or more tag filters are present -ndjson=one JSON object per line · csv=RFC 4180 with header · INT64/TIMESTAMP as decimal strings +ndjson=one object per line; INT64/TIMESTAMP strings, NULL/NaN/Inf→null · csv=RFC 4180 with header exit: 0 ok · 1 usage · 2 file open/corrupt · 3 query/runtime ``` @@ -88,7 +93,7 @@ exit: 0 ok · 1 usage · 2 file open/corrupt · 3 query/runtime B=cpp/build/Debug/bin/tsfile-cli $B meta data.tsfile; $B count -t table1 -f csv data.tsfile $B cat -t table1 --tag-filter device eq dev_1 -m temp -f ndjson data.tsfile -$B cat -m temp --start 1700000000000 -f csv data.tsfile 2>/dev/null | head +$B head -t table1 -m temp --start 1700000000000 -n 10 -f csv data.tsfile ``` ## Write @@ -107,9 +112,13 @@ COMP ∈ UNCOMPRESSED, SNAPPY, GZIP, LZO, LZ4, ZSTD, LZMA2 - CSV must contain a unique header with `time` and exactly the declared TAG/FIELD names; rows are mapped by header name, not physical column order. - `--encoding` and `--compression` are keyed by declared data type and apply to all declared columns of that type. +- CLI rejects explicit ALP, GORILLA_V1, and BOOLEAN/FLOAT/DOUBLE RLE overrides; + ALP reads work but `schema` shows `UNKNOWN`. LZMA2 requires an enabled build. +- CSV null and backslash rules: see `references/commands.md` before preparing input. - `DATE` cells are `YYYY-MM-DD`; `TIMESTAMP` and row `time` use strict decimal int64. -- **timestamps must be strictly increasing per device** (device = tag-column values); rows for - different tags may interleave/reuse timestamps. Out-of-order input → error with line number. +- **timestamps must be strictly increasing per device** (device = tag-column values); + different tags may interleave/reuse timestamps. The CLI has no duplicate-time + opt-in; out-of-order input reports its line number. - A failed import deletes its partial output; success is silent unless `-v` is used. - `-v` writes one post-commit summary to stderr: `created model=table object=
rows= output=` plus one line per column with resolved physical settings. - exit: `1` usage/parameter error · `2` CSV/input problem · `3` target/write/commit failure. @@ -125,6 +134,7 @@ Tree-model / JSON / programmatic writes → C++ SDK `cpp/examples/cpp_examples/d ## Caveats -- `head`/`cat`/`export` decode pages; nonzero exit means any produced stdout or files are not complete results. +- `head`/`cat`/`export` buffer complete output in memory, including CSV/NDJSON; + use row/time/TAG bounds for large files. Nonzero exit means incomplete results. - Table results always include all TAG columns before selected FIELD columns. - `sketch` follows the bound TsFile `printSketch` text and does not support `-f`. diff --git a/cpp/tools/skills/tsfile-cli/references/commands.md b/cpp/tools/skills/tsfile-cli/references/commands.md index 85b49203c..66cd832ef 100644 --- a/cpp/tools/skills/tsfile-cli/references/commands.md +++ b/cpp/tools/skills/tsfile-cli/references/commands.md @@ -43,4 +43,7 @@ summarizes the same supported surface for agents. `write` creates one new table-model TsFile from strict CSV. The CSV header must contain `time` and exactly the declared TAG/FIELD names. Rows are mapped by -header name. The target must not exist. +header name. The target must not exist. Unquoted `\N` is NULL; empty STRING/TEXT +cells are empty strings and quoted `"\N"` is literal text. CSV STRING/TEXT with +a leading backslash needs one extra (`\\path` imports as `\path`, quoted too). +CSV output uses the same rule; headers and non-text columns are unaffected. diff --git a/cpp/tools/skills/tsfile-cli/references/errors.md b/cpp/tools/skills/tsfile-cli/references/errors.md index 660357f03..fe15cac71 100644 --- a/cpp/tools/skills/tsfile-cli/references/errors.md +++ b/cpp/tools/skills/tsfile-cli/references/errors.md @@ -23,8 +23,8 @@ Exit status is the only success signal. - `0`: complete success. - `1`: usage or parameter error, including unknown commands, invalid options, missing scope, invalid projection, offset errors, or mismatched model/scope. -- `2`: input problem, including missing or unreadable TsFile input, damaged TsFile input, invalid CSV syntax or data during `write`, or unsupported source file type for `write -i`. -- `3`: runtime or target problem, including query/write failures, output target conflicts, output creation failure, or atomic commit failure. +- `2`: input problem, including missing or unreadable TsFile input, query setup or decode/read failures in row commands, invalid CSV syntax or data during `write`, or unsupported source file type for `write -i`. +- `3`: runtime or target problem, including output stream failures, write failures, output target conflicts, output creation failure, or atomic commit failure. Rules for callers: diff --git a/skills/tsfile-cli/SKILL.md b/skills/tsfile-cli/SKILL.md index 82c9ef94c..7abe9fb47 100644 --- a/skills/tsfile-cli/SKILL.md +++ b/skills/tsfile-cli/SKILL.md @@ -60,7 +60,7 @@ does not replace the Java batch and format-aware import tools. | `stats` | FIELD statistics, null counts, and `stats_source` | maybe | | `count` | per-column row/entity/value counts; no summary row | maybe | | `head` | first N rows (default 10, `-n`) | yes | -| `cat` | all matching rows (streamed; `table` format buffers) | yes | +| `cat` | all matching rows (buffered) | yes | | `sketch` | physical layout text with offsets and chunk/page details | no FIELD decode | | `export` | writes one object file or a numbered multi-object directory | yes | @@ -91,8 +91,9 @@ opts: -f table|ndjson|csv (default table, including pipes) - `ndjson` emits one JSON object per line. INT64/TIMESTAMP values are decimal strings; BOOLEAN/INT32/FLOAT/DOUBLE are bare values, and NULL/NaN/Inf become JSON `null`. CSV includes a header and quotes cells using RFC 4180 rules. -- The aligned `table` format buffers rows and renders control characters as - visible escapes. Prefer `csv` or `ndjson` for large dumps and pipelines. +- Row commands buffer complete output in memory, including CSV/NDJSON/export; + bound large reads with `-n` and time/TAG filters. Aligned `table` output also + uses a temporary spool and renders control characters as visible escapes. ## Sketch and export @@ -127,14 +128,20 @@ TYPE ∈ { BOOLEAN, INT32, INT64, FLOAT, DOUBLE, STRING, TEXT, TIMESTAMP, DATE, - The CSV header must contain `time` and exactly the declared TAG/FIELD names; rows are mapped by header name rather than physical column order. - `--encoding` and `--compression` may override the bound defaults by data type. +- CLI rejects explicit ALP, GORILLA_V1, and BOOLEAN/FLOAT/DOUBLE RLE overrides; + ALP reads work but `schema` shows `UNKNOWN`. LZMA2 requires an enabled build. - `--stdin` or `-i ` is required; TSV, `--columns`, `--no-header`, and `--header-match` are not supported. - Unquoted `\N` denotes NULL. Empty STRING/TEXT cells are empty strings; quoted `"\N"` is literal text. `DATE` cells are `YYYY-MM-DD`, and `TIMESTAMP`/`time` use strict decimal int64 values. +- CSV STRING/TEXT with a leading backslash needs one extra: `\\path` becomes + `\path`, including inside quotes. CSV output uses the same rule; headers + and non-text columns are unaffected. - A failed import leaves no partial output; success is silent unless `-v` is used. -- **timestamps must be strictly increasing per device** (device = tag-column values); rows for - different tags may interleave/reuse timestamps. Out-of-order input → error with line number. +- **timestamps must be strictly increasing per device** (device = tag-column values); + different tags may interleave/reuse timestamps. The CLI has no duplicate-time + opt-in; out-of-order input reports its line number. - `-v` writes a post-commit summary and resolved column physical settings to stderr. Tree-model / JSON / programmatic writes → C++ SDK `cpp/examples/cpp_examples/demo_write.cpp` diff --git a/skills/tsfile/SKILL.md b/skills/tsfile/SKILL.md index 21768ad67..05cb900d4 100644 --- a/skills/tsfile/SKILL.md +++ b/skills/tsfile/SKILL.md @@ -33,7 +33,7 @@ description: >- Use this skill for SDK code, Java import and point-count tools, Tree/Table model decisions, schema design, compatibility, and cross-language integration. Use the sibling `../tsfile-cli/SKILL.md` for the C++ `tsfile-cli` binary: file -inspection, preview, export, sampling, and its explicit single-table CSV +inspection, preview, export, and its explicit single-table CSV `write` operation. ## Operating Rules @@ -66,18 +66,23 @@ Read only the files required by the current task: - Java SDK code and API guardrails: `references/java.md` - Java CSV/Parquet/Arrow import and table point-count tools: `references/java-tools.md` -- Python DataFrame datasets, pandas conversion, and Arrow batch I/O: +- Python DataFrame datasets, persistent indexes, resource limits, pandas + conversion, and Arrow batch I/O: `references/dataframe-arrow.md` -- File-level properties and Table-model TAG filters across bindings: +- File-level properties, timestamp-policy metadata, and Table-model TAG + filters across bindings: `references/properties-filters.md` -- C++ static/shared linking and parallel write configuration: +- C++ static/shared and installed SDK linking, plus parallel read/write + configuration: `references/cpp-build-write.md` - Current Java and C++ encoding/compression factory matrices: `references/encoding-compression.md` -- Query-by-row, timeseries metadata, C++ incomplete-file recovery and append, - Java TsBlock writes, custom/buffered input, and Tablet object/size APIs: +- Query-by-row, timestamp ordering, timeseries metadata, C++ random-access + sources/sinks and incomplete-file recovery, Java TsBlock writes, + custom/buffered input, and Tablet object/size APIs: `references/advanced-apis.md` -- Python SDK code and binding-specific behavior: `references/python.md` +- Python SDK code, seekable sources, local read backends, and parallel controls: + `references/python.md` - C++ SDK code and resource management: `references/cpp.md` - C wrapper entry points and lifecycle: `references/c.md` - Version resolution, build requirements, and cross-version checks: @@ -104,10 +109,11 @@ or language bindings. ## Current Baseline Routing - Use `references/dataframe-arrow.md` for `TsFileDataFrame`, pandas conversion, - PyArrow `Table`/`RecordBatch`, or Arrow C Data batch paths. + persistent indexes and resource limits, PyArrow `Table`/`RecordBatch`, or + Arrow C Data batch paths. - Use `references/properties-filters.md` for binary file properties or Table-model TAG predicate construction and composition. -- Use `references/cpp-build-write.md` for C++ library linkage or write +- Use `references/cpp-build-write.md` for C++ library linkage or read/write concurrency. Configure the current source APIs; do not emit legacy aliases. - Use `references/encoding-compression.md` before naming a factory or claiming that an encoding/type or compressor combination is constructible. diff --git a/skills/tsfile/references/advanced-apis.md b/skills/tsfile/references/advanced-apis.md index c8672657f..90e32b273 100644 --- a/skills/tsfile/references/advanced-apis.md +++ b/skills/tsfile/references/advanced-apis.md @@ -31,6 +31,18 @@ the listed current source. accept `tag_filter` and `batch_size`. - Treat `limit < 0` as unlimited and reject negative offsets. Close or destroy every result set through the binding that created it. +- Table `TsFileReader::query` also has an overload combining start/end times, + offset/limit, a TAG filter, and batch size; `queryByRow` has no time bounds. + +## Timestamp Ordering + +Strict order is default; equal timestamps can be preserved as separate rows. +Before writer creation, use Java config `setAllowNonDecreasingTimestamps(true)`, +C++ `common::set_allow_non_decreasing_timestamps(true)` (C: without `common::`), +or Python writer argument `allow_non_decreasing_timestamps=True` (default `False`). +This is process-wide, including Python constructor defaults; avoid conflicting +writers. Decreases remain invalid. See `properties-filters.md` for the persisted +policy; CLI `write` has no opt-in. ## Timeseries Metadata @@ -57,7 +69,7 @@ the current `TableSchema`; do not infer constructor positions from memory. - Use `Tablet.serializedSize()` to allocate or validate the serialized Tablet payload. Do not replace it with a RAM-size estimate. -## Custom and Buffered Java Input +## Custom and Buffered Input `TsFileSequenceReader` accepts a `TsFileInput`, enabling custom storage/input implementations. For local buffered reads, construct `BufferedTsFileInput(Path, @@ -65,6 +77,14 @@ bufferSize)` and pass it to the reader. Buffer size must be positive, and the reader owns and closes the supplied input if initialization fails or when the reader is closed. +C++ `TsFileReader::open(unique_ptr)` owns its source; +custom sources must handle short/concurrent reads. Writers borrow an open +`RandomAccessWriteFile*` through closure. For in-memory readback, move the closed +`MemoryRandomAccessWriteFile::release_data()` into `MemoryRandomAccessReadFile`. +`common::set_file_read_backend(common::FileReadBackend::MMAP)` affects new local +opens: PREAD is default, MMAP requires mapping, AUTO allows PREAD fallback. +Do not mutate mapped files; Python file objects (`python.md`) bypass this setting. + ## C++ Recovery and Append Use `storage::RestorableTsFileIOWriter` to inspect and optionally truncate an @@ -95,5 +115,8 @@ tests. Ranges are `(start, length)`, not `(start, end)`; empty ranges return `java/tsfile/src/main/java/org/apache/tsfile/read/reader/` - Recovery: `cpp/src/file/restorable_tsfile_io_writer.h` and the current C++ Tree/Table writer headers and recovery tests +- Timestamp/backend config: `cpp/src/common/global.h`, Java `TSFileConfig.java`, + and the Python writer bindings +- Random access: `cpp/src/file/random_access_*_file.h` and `memory_random_access_*_file.h` - BitMap: `java/common/src/main/java/org/apache/tsfile/utils/BitMap.java` and `java/tsfile/src/test/java/org/apache/tsfile/utils/BitMapTest.java` diff --git a/skills/tsfile/references/c.md b/skills/tsfile/references/c.md index 843de32c6..ceba30afc 100644 --- a/skills/tsfile/references/c.md +++ b/skills/tsfile/references/c.md @@ -34,6 +34,13 @@ Use this reference only for the C wrapper around the C++ implementation. Read typed `tablet_add_value_by_name_*` functions. - Write: `tsfile_writer_write`. - Reader/query: `tsfile_reader_new`, `tsfile_query_table`. +- Runtime: `tsfile_set_parallel_read_enabled`, `tsfile_set_parallel_write_enabled`, + `tsfile_set_thread_count` and matching getters; defaults enabled, enabled, 6. + Counts must be `1..64`; do not resize an in-use pool (see `cpp-build-write.md`). +- Local reads: `tsfile_set_file_read_backend` / `tsfile_get_file_read_backend`; + use the declared `TSFILE_READ_BACKEND_*` values (see `advanced-apis.md`). +- Timestamp policy: `set_allow_non_decreasing_timestamps(bool)` (no `tsfile_` + prefix), default false; configure before creating writers (see `advanced-apis.md`). - Cleanup: use the matching close and `free_*` functions demonstrated by the maintained examples. diff --git a/skills/tsfile/references/compatibility.md b/skills/tsfile/references/compatibility.md index d261548fa..72165f436 100644 --- a/skills/tsfile/references/compatibility.md +++ b/skills/tsfile/references/compatibility.md @@ -68,6 +68,8 @@ Repository checks: null, time, text, date, and binary values. 6. Close writers, readers, and result sets to finalize file structures and release native resources. +7. Java cannot query ALP series. Duplicate timestamps are opt-in; check the + file's timestamp-policy property and the consumer's ordering requirements. ## Troubleshooting Order diff --git a/skills/tsfile/references/cpp-build-write.md b/skills/tsfile/references/cpp-build-write.md index 87d8d7307..52b9dd7c8 100644 --- a/skills/tsfile/references/cpp-build-write.md +++ b/skills/tsfile/references/cpp-build-write.md @@ -17,7 +17,7 @@ under the License. --> -# C++ Build and Parallel Write +# C++ Build and Parallel I/O ## Shared or Static libtsfile @@ -42,11 +42,17 @@ consumer of the static archive must define `TSFILE_STATIC`; consumers of the exported CMake target inherit the required definition. Verify the artifact in `cpp/target/build/lib` for Maven or the selected CMake build tree. +Installed SDKs export `TsFile::tsfile` via `find_package(TsFile CONFIG REQUIRED)`; +set `CMAKE_PREFIX_PATH` to the prefix. Unix consumers can use +`pkg-config --cflags --libs libtsfile` (`--static` for static linkage). +Match architecture/MSVC runtime and retain required system codec packages; +see `packaging/README.md`. `build.sh` installs only with explicit `install`. + `ENABLE_SIMD` also defaults to `ON` in the current CMake and Maven builds. Set it explicitly only for portability diagnosis or controlled benchmarking; it is independent of static/shared selection and thread support. -## Parallel Write +## Parallel Read and Write Compile thread support with `ENABLE_THREADS=ON` (the default). `OFF` strips the threaded paths. Configure the current global worker pool through @@ -55,20 +61,17 @@ threaded paths. Configure the current global worker pool through ```cpp #include "common/global.h" +common::set_parallel_read_enabled(false); common::set_parallel_write_enabled(true); int status = common::set_thread_count(4); // valid range: 1..64 ``` -The current baseline uses `common::set_parallel_write_enabled` and -`common::set_thread_count`; use those exact names and namespace. The pool is -shared by parallel read and write paths. Configure it before creating active -readers/writers. Changing the count after initialization rebuilds the pool and -must never race with an operation using it. - -Current source initializes parallel writing as enabled and the pool size as 6. -Do not substitute a hardware-concurrency default from older prose. Parallel -work is used only when the compiled path and runtime conditions qualify; keep -the tablet/batch write contract correct when it falls back to serial work. +Both flags default to enabled, have matching `get_parallel_*_enabled()` getters, +and apply to subsequent operations. Configure before queries/writes; never +resize the shared pool while a reader, writer, or result set is in use. +The pool defaults to 6 threads, not hardware concurrency. Check setter return +codes; failed replacement preserves the old pool. Parallel execution still +depends on the compiled path and workload; preserve serial-path correctness. Always check write return codes. If a multi-column batch fails partway, stop using that writer rather than attempting to flush a potentially misaligned @@ -78,6 +81,8 @@ partial batch. - Link selection: `cpp/CMakeLists.txt`, `cpp/src/CMakeLists.txt`, `cpp/build.sh`, `cpp/pom.xml`, and `cpp/README.md` +- Installed SDK: `cpp/cmake/TsFileConfig.cmake.in`, + `cpp/cmake/libtsfile.pc.in`, and `packaging/README.md` - Runtime API and defaults: `cpp/src/common/global.h`, `cpp/src/common/global.cc`, and `cpp/src/common/config/config.h` - Write behavior: `cpp/src/writer/tsfile_writer.cc` and current writer tests diff --git a/skills/tsfile/references/dataframe-arrow.md b/skills/tsfile/references/dataframe-arrow.md index 843c00def..d82734242 100644 --- a/skills/tsfile/references/dataframe-arrow.md +++ b/skills/tsfile/references/dataframe-arrow.md @@ -70,6 +70,15 @@ subset view to close shared readers. Boolean dataset selection is positional: build the mask from `ds[...]`, as above; use `list_timeseries_metadata()` for named inspection rather than as the mask source. +`use_index=True` builds/reuses an index (default `False`). Indexed mode assumes +immutable sources (`trust_index=True`); use `False` for generation checks and +stale-index rebuilds. It also accepts `max_prepared_series`, `descriptor_cache_size`, +`max_open_files`, `query_workers`, and `query_parallel_min_rows`; explicit values +override `TSFILE_DATAFRAME_` then defaults. Subsets share limits; +cache limits allow zero, others must be positive. Active queries may pin cache +entries above these entry-count limits. `query_workers=1` serializes dataset +groups, independently of the native pool. See `python/README.md` for details. + ## Arrow Batch I/O ```python @@ -97,4 +106,5 @@ possible so it resolves the time-column index from the schema. reader/writer bindings - `python/tsfile/tsfile_table_writer.py` - `python/tests/test_dataframe.py`, `python/tests/test_tsfile_dataset.py`, - `python/tests/test_batch_arrow.py`, and `python/tests/test_write_arrow.py` + `python/tests/test_dataset_read_options.py`, `python/tests/test_batch_arrow.py`, + and `python/tests/test_write_arrow.py` diff --git a/skills/tsfile/references/docs-map.yaml b/skills/tsfile/references/docs-map.yaml index 51153b215..8b0ab4470 100644 --- a/skills/tsfile/references/docs-map.yaml +++ b/skills/tsfile/references/docs-map.yaml @@ -16,7 +16,7 @@ # under the License. schema_version: 1 -last_verified: "2026-08-17" +last_verified: "2026-11-15" policy_reference: references/source-policy.md path_bases: policy_reference: skill_root @@ -25,6 +25,7 @@ path_bases: skill_baseline: scope: current-source source_ref: develop + source_commit: "b16df6b087d8a68f5921a3bcd3714abe3cf00a1e" maven_project_version: "2.5.1-SNAPSHOT" java_release: "17" cpp_sdk_version: "2.5.1.dev" @@ -115,11 +116,14 @@ dataframe: local_authorities: - python/tsfile/utils.py - python/tsfile/dataset/ + - python/tsfile/dataset/_config.py + - python/tsfile/dataset/runtime.py - python/tsfile/tsfile_reader.pyx - python/tsfile/tsfile_writer.pyx - python/tsfile/tsfile_table_writer.py - python/tests/test_dataframe.py - python/tests/test_tsfile_dataset.py + - python/tests/test_dataset_read_options.py - python/tests/test_batch_arrow.py - python/tests/test_write_arrow.py - skills/tsfile/references/dataframe-arrow.md @@ -134,6 +138,8 @@ current_capabilities: - cpp/src/reader/tsfile_reader.h - cpp/src/writer/tsfile_writer.h - cpp/src/writer/tsfile_table_writer.h + - cpp/src/common/constant/tsfile_constant.h + - java/tsfile/src/main/java/org/apache/tsfile/common/conf/TSFileConfig.java - cpp/src/cwrapper/tsfile_cwrapper.h - python/tsfile/tag_filter.py - python/tsfile/tsfile_reader.pyx @@ -149,6 +155,9 @@ current_capabilities: - cpp/src/common/global.cc - cpp/src/common/config/config.h - cpp/src/writer/tsfile_writer.cc + - cpp/src/cwrapper/tsfile_cwrapper.h + - python/tsfile/tsfile_py_cpp.pyx + - packaging/README.md - skills/tsfile/references/cpp-build-write.md encoding_and_compression_factories: local_authorities: @@ -176,6 +185,11 @@ current_capabilities: - java/tsfile/src/main/java/org/apache/tsfile/read/reader/TsFileInput.java - java/tsfile/src/main/java/org/apache/tsfile/read/reader/BufferedTsFileInput.java - cpp/src/file/restorable_tsfile_io_writer.h + - cpp/src/file/random_access_read_file.h + - cpp/src/file/random_access_write_file.h + - cpp/src/file/memory_random_access_read_file.h + - cpp/src/file/memory_random_access_write_file.h + - cpp/src/common/global.h - cpp/src/writer/tsfile_tree_writer.h - cpp/src/writer/tsfile_table_writer.h - java/common/src/main/java/org/apache/tsfile/utils/BitMap.java diff --git a/skills/tsfile/references/encoding-compression.md b/skills/tsfile/references/encoding-compression.md index b745687ca..010f83944 100644 --- a/skills/tsfile/references/encoding-compression.md +++ b/skills/tsfile/references/encoding-compression.md @@ -54,13 +54,16 @@ Current time factories accept `PLAIN` or `TS_2DIFF`. |---|---| | `PLAIN` | all types handled by `PlainEncoder`/`PlainDecoder` | | `DICTIONARY` | `STRING`, `TEXT` | -| `RLE` | `INT32`, `DATE`, `INT64`, `TIMESTAMP` | -| `TS_2DIFF`, `GORILLA` | `INT32`, `DATE`, `INT64`, `TIMESTAMP`, `FLOAT`, `DOUBLE` | -| `ZIGZAG` | `INT32`, `INT64` | -| `SPRINTZ` | `INT32`, `INT64`, `FLOAT`, `DOUBLE` | +| `RLE` | `BOOLEAN`, `INT32`, `DATE`, `INT64`, `TIMESTAMP`, `FLOAT`, `DOUBLE` | +| `TS_2DIFF`, `GORILLA`, `CHIMP`, `SPRINTZ`, `RLBE` | `INT32`, `DATE`, `INT64`, `TIMESTAMP`, `FLOAT`, `DOUBLE` | +| `ZIGZAG` | `INT32`, `DATE`, `INT64`, `TIMESTAMP` | +| `GORILLA_V1`, `ALP` | `FLOAT`, `DOUBLE` | +| `CAMEL` | `DOUBLE` | `CompressorFactory::alloc_compressor` always supports `UNCOMPRESSED`; `SNAPPY`, -`GZIP`, `LZO`, and `LZ4` require their corresponding compile definitions. +`GZIP`, `LZO`, `LZ4`, `ZSTD`, and `LZMA2` require their corresponding compile +definitions. LZMA2 is opt-in (`ENABLE_LZMA2=ON`); check the target build rather +than assuming every enum is enabled. `SDT`, `PAA`, and `PLA` are enum values but the current compressor factory returns no implementation for them. Check for `nullptr` and release successful allocations with `CompressorFactory::free`. @@ -68,6 +71,9 @@ allocations with `CompressorFactory::free`. Python exposes schema encoding/compression enums through the native binding but does not expose these Java/C++ low-level factories as an equivalent Python API. +ALP is C++/Python-only; Java cannot query ALP series. CLI write support is +narrower than this SDK matrix; use the `tsfile-cli` skill for its restrictions. + ## Source Anchors - Java: `TSEncoding.java`, `TSEncodingBuilder.java`, `Decoder.java`, diff --git a/skills/tsfile/references/java-tools.md b/skills/tsfile/references/java-tools.md index e6b0b2116..2c094bffe 100644 --- a/skills/tsfile/references/java-tools.md +++ b/skills/tsfile/references/java-tools.md @@ -74,9 +74,10 @@ resolve the exact input path, ensure the containing directory is writable and has room for a complete temporary copy, and preserve a backup when the data is not otherwise recoverable. Missing and incomplete files must remain errors. -Do not use C++ `tsfile-cli count` as a substitute. That command reads -per-series statistics and prints counts; it neither validates nor writes the -table-level point-count property. +Do not use C++ `tsfile-cli count` as a substitute. That command reports +per-column logical counts, scans rows in the Table model, and may also scan +when metadata is insufficient. It neither validates nor writes the table-level +point-count property. ## Source Anchors diff --git a/skills/tsfile/references/properties-filters.md b/skills/tsfile/references/properties-filters.md index 6fd461d24..4bd8d4378 100644 --- a/skills/tsfile/references/properties-filters.md +++ b/skills/tsfile/references/properties-filters.md @@ -40,6 +40,10 @@ The Java table point-count metadata is a reserved use of file properties. Use `references/java-tools.md` for inspection or backfill instead of editing those keys directly. +Writers with duplicate timestamps enabled record `allowNonDecreasingTimestamps` +as UTF-8 `true`. Configure the writer policy in `advanced-apis.md`; setting +the property manually does not enable it. + ## Table TAG Filters Build TAG predicates against a `TableSchema`, apply them only to Table-model diff --git a/skills/tsfile/references/python.md b/skills/tsfile/references/python.md index 715ed6f58..24a9b571b 100644 --- a/skills/tsfile/references/python.md +++ b/skills/tsfile/references/python.md @@ -73,6 +73,20 @@ with TsFileReader("data.tsfile") as reader: ## Guidance +- Writers accept `allow_non_decreasing_timestamps=False`; this is process-wide. + See `advanced-apis.md` for opt-in semantics. +- For persistent dataset indexes and bounded caches, see `dataframe-arrow.md`. +- Parallel controls: `set_parallel_read_enabled`, `set_parallel_write_enabled`, + `set_thread_count` and matching `get_*` functions. Flags default to `True`, + pool size to 6; counts outside `1..64` raise `InvalidArgumentError`. + Configure before operations and never resize an in-use pool. Equivalent + config keys: `parallel_read_enabled_`, `parallel_write_enabled_`, `thread_count_`. +- `TsFileReader` also accepts binary objects with `seek`, `tell`, and `read(size)` + (e.g. fsspec streams); it retains the object, preserves its cursor, and does + not close it. The caller owns the source. +- For local paths, `set_file_read_backend(FileReadBackend.MMAP)` + affects subsequently opened files; PREAD is default. See `advanced-apis.md` + for backend semantics. File objects do not use this setting. - Current bindings expose `query_table` and `query_table_by_row`; do not reuse older `get_table_names` or `read_table` examples without checking the target version.