Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 24 additions & 17 deletions cpp/tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,11 +96,11 @@ Exit codes: `0` success, `1` usage/argument error, `2` file open/corrupt,
| `count` | Object/column counts; no synthetic summary row |
| `sketch` | Print the physical file sketch, optionally to `-o` |
| `head` | First N rows (default 10; use `-n`) |
| `cat` | All matching rows, streamed (`table` format buffers to align columns) |
| `cat` | All matching rows, buffered before publishing |
| `export` | Export one object to `-o`, or multiple objects to `--output-dir`, using `--type` |

The metadata commands (`ls` / `schema` / `meta` / `stats` / `count`) answer most questions
without decoding data pages.
Use `ls`, `schema`, and `meta` for metadata-only discovery. `stats` can scan
when statistics are unavailable, and table-model `count` scans rows.

Shared options:

Expand All @@ -114,14 +114,17 @@ Shared options:
| `--tag-filter C OP [V]` | Table TAG predicate for row reads; `OP` is `eq`, `neq`, `regexp`, `is-null`, or `not-null` |
| `--tag-match all\|any` | Required when more than one `--tag-filter` is supplied |

`ndjson` output emits one JSON object per line; numbers/booleans are bare, other values are
quoted, nulls are `null`, and non-finite floats become `null`. CSV output follows RFC 4180.
`ndjson` output emits one JSON object per line; INT64/TIMESTAMP values are decimal
strings, BOOLEAN/INT32/FLOAT/DOUBLE are bare values, nulls are `null`, and
non-finite floats become `null`. CSV output follows RFC 4180.
CSV nulls are unquoted `\N`; empty strings are quoted as `""`. Non-null STRING/TEXT
values beginning with `\` gain one extra leading `\`, using the same encoding rule
as CSV input described below. Timestamps are raw int64 values. The `table` format
uses a temporary spool to align columns with bounded memory; prefer `csv`/`ndjson`
when temporary disk use is undesirable. `sketch`
does not accept `--format`.
as CSV input described below. Timestamps are raw int64 values. Row commands
currently stage the complete formatted result in memory before publishing it,
including CSV/NDJSON and `export`. The `table` formatter additionally uses a
temporary spool to align columns; this does not bound memory for the complete
output path. Use row, time, and TAG bounds for large reads. `sketch` does not
accept `--format`.

Text output preserves well-formed UTF-8 and replaces malformed UTF-8 in names
and text with U+FFFD (`�`). This also applies to sketch output and export
Expand Down Expand Up @@ -165,6 +168,11 @@ By default each column uses the engine's default encoding and compression for it
`--encoding` and `--compression` override by canonical data type, applying to every declared
TAG/FIELD of that type; they do not target individual columns and do not affect `time`.

The CLI currently rejects explicit ALP, GORILLA_V1, and BOOLEAN/FLOAT/DOUBLE RLE
overrides even though the SDK supports them. ALP files can be read, but `schema`
currently displays their encoding as `UNKNOWN`. Native build options also
control compression availability; LZMA2 is opt-in.

| Option | Meaning |
|---|---|
| `--table <name>` | Output table name (lower-cased) |
Expand Down Expand Up @@ -226,18 +234,17 @@ the skill just means placing it there — either project-level or user-level:

```bash
# project-level (this repository only)
mkdir -p .claude/skills/tsfile-cli
cp cpp/tools/skills/tsfile-cli/SKILL.md .claude/skills/tsfile-cli/SKILL.md
mkdir -p .claude/skills
cp -R cpp/tools/skills/tsfile-cli .claude/skills/

# or user-level (available in all your projects)
mkdir -p ~/.claude/skills/tsfile-cli
cp cpp/tools/skills/tsfile-cli/SKILL.md ~/.claude/skills/tsfile-cli/SKILL.md
mkdir -p ~/.claude/skills
cp -R cpp/tools/skills/tsfile-cli ~/.claude/skills/
```

> The installed `SKILL.md` must begin with its YAML front-matter (`--- … ---`) for the
> assistant to detect it. The in-repo copy carries an Apache license header comment above
> the front-matter; if discovery fails, delete that leading `<!-- … -->` block from the
> installed copy so `---` is the first line.
Copy the entire directory: `SKILL.md` loads the bundled `references/` files.
The current entrypoint already begins with YAML front-matter and keeps its
Apache license as YAML comments; no header removal is needed.

Start a new assistant session afterward. The skill then activates automatically when you
ask to inspect or import a `.tsfile`; you can also invoke it explicitly (e.g. "use the
Expand Down
36 changes: 23 additions & 13 deletions cpp/tools/skills/tsfile-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
name: tsfile-cli
description: >-
Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect,
preview, export, or create an Apache TsFile; report metadata or per-series
preview, export, or create an Apache TsFile; report metadata or per-column
counts; or use its explicit single-table CSV write command.
---

Expand All @@ -41,14 +41,17 @@ Use this skill only for the C++ `tsfile-cli` binary. Route Java
checks or backfill to the top-level `tsfile` skill when it is installed.

The names overlap but the semantics do not: `tsfile-cli count` is a read-only
per-series report, not the Java table point-count property tool. `tsfile-cli
per-column report, not the Java table point-count property tool. `tsfile-cli
write` is the C++ binary's narrow one-file/stream, one-table CSV import; it
does not replace the Java batch and format-aware import tools.

## Binary

- Name `tsfile-cli` (CMake target `tsfile_cli`). Find: `ls cpp/build/*/bin/tsfile-cli`.
- Name `tsfile-cli` (CMake target `tsfile_cli`). Check `PATH`,
`cpp/build/*/bin/tsfile-cli`, or `cpp/target/build/bin/tsfile-cli`.
- Build only if missing: `cd cpp && bash build.sh -t=Debug`.
- Use `tsfile-cli <command> --help` for the installed binary's supported
syntax and result fields before choosing flags.

## Read

Expand All @@ -62,33 +65,35 @@ does not replace the Java batch and format-aware import tools.
| `stats` | FIELD statistics with null counts and source | maybe |
| `count` | object/column counts; no summary row | maybe |
| `head` | first N rows (default 10, `-n`) | yes |
| `cat` | all matching rows (streamed; `table` format buffers) | yes |
| `cat` | all matching rows (buffered) | yes |
| `sketch` | physical layout text from the bound `printSketch` behavior | no FIELD decode |
| `export` | writes one object file or a numbered multi-object directory | yes |

Prefer metadata verbs (`ls/schema/meta/stats/count`) before row verbs so the
file model, scope, and columns are known.
Inspect `ls`, `schema`, and `meta` before querying rows to identify the model,
objects, and columns. `stats` can scan when statistics are unavailable, and
table-model `count` scans rows; neither is guaranteed to be metadata-only.

For `head` and `cat`, omit `-d/-t` only when the selected model has exactly one
accessible object. Multi-object files require explicit scope. `export` always
accessible object. Multi-object files require explicit scope. `schema`,
`stats`, and `count` visit every object when scope is omitted. `export` always
requires explicit scope.

```
opts: -f table|ndjson|csv (default table)
opts: -f table|ndjson|csv (default table, including pipes)
-d <device> | -t <table> (mutually exclusive)
-m <column> repeat for projection or metadata filtering; no comma lists
-n N · --offset N · --start <int64> · --end <int64> for head/cat/export
--tag-filter C OP [V] where OP=eq|neq|regexp|is-null|not-null
--tag-match all|any when two or more tag filters are present
ndjson=one JSON object per line · csv=RFC 4180 with header · INT64/TIMESTAMP as decimal strings
ndjson=one object per line; INT64/TIMESTAMP strings, NULL/NaN/Inf→null · csv=RFC 4180 with header
exit: 0 ok · 1 usage · 2 file open/corrupt · 3 query/runtime
```

```sh
B=cpp/build/Debug/bin/tsfile-cli
$B meta data.tsfile; $B count -t table1 -f csv data.tsfile
$B cat -t table1 --tag-filter device eq dev_1 -m temp -f ndjson data.tsfile
$B cat -m temp --start 1700000000000 -f csv data.tsfile 2>/dev/null | head
$B head -t table1 -m temp --start 1700000000000 -n 10 -f csv data.tsfile
```

## Write
Expand All @@ -107,9 +112,13 @@ COMP ∈ UNCOMPRESSED, SNAPPY, GZIP, LZO, LZ4, ZSTD, LZMA2

- CSV must contain a unique header with `time` and exactly the declared TAG/FIELD names; rows are mapped by header name, not physical column order.
- `--encoding` and `--compression` are keyed by declared data type and apply to all declared columns of that type.
- CLI rejects explicit ALP, GORILLA_V1, and BOOLEAN/FLOAT/DOUBLE RLE overrides;
ALP reads work but `schema` shows `UNKNOWN`. LZMA2 requires an enabled build.
- CSV null and backslash rules: see `references/commands.md` before preparing input.
- `DATE` cells are `YYYY-MM-DD`; `TIMESTAMP` and row `time` use strict decimal int64.
- **timestamps must be strictly increasing per device** (device = tag-column values); rows for
different tags may interleave/reuse timestamps. Out-of-order input → error with line number.
- **timestamps must be strictly increasing per device** (device = tag-column values);
different tags may interleave/reuse timestamps. The CLI has no duplicate-time
opt-in; out-of-order input reports its line number.
- A failed import deletes its partial output; success is silent unless `-v` is used.
- `-v` writes one post-commit summary to stderr: `created model=table object=<table> rows=<n> output=<path>` plus one line per column with resolved physical settings.
- exit: `1` usage/parameter error · `2` CSV/input problem · `3` target/write/commit failure.
Expand All @@ -125,6 +134,7 @@ Tree-model / JSON / programmatic writes → C++ SDK `cpp/examples/cpp_examples/d

## Caveats

- `head`/`cat`/`export` decode pages; nonzero exit means any produced stdout or files are not complete results.
- `head`/`cat`/`export` buffer complete output in memory, including CSV/NDJSON;
use row/time/TAG bounds for large files. Nonzero exit means incomplete results.
- Table results always include all TAG columns before selected FIELD columns.
- `sketch` follows the bound TsFile `printSketch` text and does not support `-f`.
5 changes: 4 additions & 1 deletion cpp/tools/skills/tsfile-cli/references/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,7 @@ summarizes the same supported surface for agents.

`write` creates one new table-model TsFile from strict CSV. The CSV header must
contain `time` and exactly the declared TAG/FIELD names. Rows are mapped by
header name. The target must not exist.
header name. The target must not exist. Unquoted `\N` is NULL; empty STRING/TEXT
cells are empty strings and quoted `"\N"` is literal text. CSV STRING/TEXT with
a leading backslash needs one extra (`\\path` imports as `\path`, quoted too).
CSV output uses the same rule; headers and non-text columns are unaffected.
4 changes: 2 additions & 2 deletions cpp/tools/skills/tsfile-cli/references/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ Exit status is the only success signal.

- `0`: complete success.
- `1`: usage or parameter error, including unknown commands, invalid options, missing scope, invalid projection, offset errors, or mismatched model/scope.
- `2`: input problem, including missing or unreadable TsFile input, damaged TsFile input, invalid CSV syntax or data during `write`, or unsupported source file type for `write -i`.
- `3`: runtime or target problem, including query/write failures, output target conflicts, output creation failure, or atomic commit failure.
- `2`: input problem, including missing or unreadable TsFile input, query setup or decode/read failures in row commands, invalid CSV syntax or data during `write`, or unsupported source file type for `write -i`.
- `3`: runtime or target problem, including output stream failures, write failures, output target conflicts, output creation failure, or atomic commit failure.

Rules for callers:

Expand Down
17 changes: 12 additions & 5 deletions skills/tsfile-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ does not replace the Java batch and format-aware import tools.
| `stats` | FIELD statistics, null counts, and `stats_source` | maybe |
| `count` | per-column row/entity/value counts; no summary row | maybe |
| `head` | first N rows (default 10, `-n`) | yes |
| `cat` | all matching rows (streamed; `table` format buffers) | yes |
| `cat` | all matching rows (buffered) | yes |
| `sketch` | physical layout text with offsets and chunk/page details | no FIELD decode |
| `export` | writes one object file or a numbered multi-object directory | yes |

Expand Down Expand Up @@ -91,8 +91,9 @@ opts: -f table|ndjson|csv (default table, including pipes)
- `ndjson` emits one JSON object per line. INT64/TIMESTAMP values are decimal
strings; BOOLEAN/INT32/FLOAT/DOUBLE are bare values, and NULL/NaN/Inf become
JSON `null`. CSV includes a header and quotes cells using RFC 4180 rules.
- The aligned `table` format buffers rows and renders control characters as
visible escapes. Prefer `csv` or `ndjson` for large dumps and pipelines.
- Row commands buffer complete output in memory, including CSV/NDJSON/export;
bound large reads with `-n` and time/TAG filters. Aligned `table` output also
uses a temporary spool and renders control characters as visible escapes.

## Sketch and export

Expand Down Expand Up @@ -127,14 +128,20 @@ TYPE ∈ { BOOLEAN, INT32, INT64, FLOAT, DOUBLE, STRING, TEXT, TIMESTAMP, DATE,
- The CSV header must contain `time` and exactly the declared TAG/FIELD names;
rows are mapped by header name rather than physical column order.
- `--encoding` and `--compression` may override the bound defaults by data type.
- CLI rejects explicit ALP, GORILLA_V1, and BOOLEAN/FLOAT/DOUBLE RLE overrides;
ALP reads work but `schema` shows `UNKNOWN`. LZMA2 requires an enabled build.
- `--stdin` or `-i <input.csv>` is required; TSV, `--columns`, `--no-header`, and
`--header-match` are not supported.
- Unquoted `\N` denotes NULL. Empty STRING/TEXT cells are empty strings;
quoted `"\N"` is literal text. `DATE` cells are `YYYY-MM-DD`, and
`TIMESTAMP`/`time` use strict decimal int64 values.
- CSV STRING/TEXT with a leading backslash needs one extra: `\\path` becomes
`\path`, including inside quotes. CSV output uses the same rule; headers
and non-text columns are unaffected.
- A failed import leaves no partial output; success is silent unless `-v` is used.
- **timestamps must be strictly increasing per device** (device = tag-column values); rows for
different tags may interleave/reuse timestamps. Out-of-order input → error with line number.
- **timestamps must be strictly increasing per device** (device = tag-column values);
different tags may interleave/reuse timestamps. The CLI has no duplicate-time
opt-in; out-of-order input reports its line number.
- `-v` writes a post-commit summary and resolved column physical settings to stderr.

Tree-model / JSON / programmatic writes → C++ SDK `cpp/examples/cpp_examples/demo_write.cpp`
Expand Down
24 changes: 15 additions & 9 deletions skills/tsfile/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ description: >-
Use this skill for SDK code, Java import and point-count tools, Tree/Table model
decisions, schema design, compatibility, and cross-language integration. Use
the sibling `../tsfile-cli/SKILL.md` for the C++ `tsfile-cli` binary: file
inspection, preview, export, sampling, and its explicit single-table CSV
inspection, preview, export, and its explicit single-table CSV
`write` operation.

## Operating Rules
Expand Down Expand Up @@ -66,18 +66,23 @@ Read only the files required by the current task:
- Java SDK code and API guardrails: `references/java.md`
- Java CSV/Parquet/Arrow import and table point-count tools:
`references/java-tools.md`
- Python DataFrame datasets, pandas conversion, and Arrow batch I/O:
- Python DataFrame datasets, persistent indexes, resource limits, pandas
conversion, and Arrow batch I/O:
`references/dataframe-arrow.md`
- File-level properties and Table-model TAG filters across bindings:
- File-level properties, timestamp-policy metadata, and Table-model TAG
filters across bindings:
`references/properties-filters.md`
- C++ static/shared linking and parallel write configuration:
- C++ static/shared and installed SDK linking, plus parallel read/write
configuration:
`references/cpp-build-write.md`
- Current Java and C++ encoding/compression factory matrices:
`references/encoding-compression.md`
- Query-by-row, timeseries metadata, C++ incomplete-file recovery and append,
Java TsBlock writes, custom/buffered input, and Tablet object/size APIs:
- Query-by-row, timestamp ordering, timeseries metadata, C++ random-access
sources/sinks and incomplete-file recovery, Java TsBlock writes,
custom/buffered input, and Tablet object/size APIs:
`references/advanced-apis.md`
- Python SDK code and binding-specific behavior: `references/python.md`
- Python SDK code, seekable sources, local read backends, and parallel controls:
`references/python.md`
- C++ SDK code and resource management: `references/cpp.md`
- C wrapper entry points and lifecycle: `references/c.md`
- Version resolution, build requirements, and cross-version checks:
Expand All @@ -104,10 +109,11 @@ or language bindings.
## Current Baseline Routing

- Use `references/dataframe-arrow.md` for `TsFileDataFrame`, pandas conversion,
PyArrow `Table`/`RecordBatch`, or Arrow C Data batch paths.
persistent indexes and resource limits, PyArrow `Table`/`RecordBatch`, or
Arrow C Data batch paths.
- Use `references/properties-filters.md` for binary file properties or
Table-model TAG predicate construction and composition.
- Use `references/cpp-build-write.md` for C++ library linkage or write
- Use `references/cpp-build-write.md` for C++ library linkage or read/write
concurrency. Configure the current source APIs; do not emit legacy aliases.
- Use `references/encoding-compression.md` before naming a factory or claiming
that an encoding/type or compressor combination is constructible.
Expand Down
Loading