Skip to content

AGENTS.md

Latest commit

 

History

History
91 lines (78 loc) · 4.69 KB

AGENTS.md

File metadata and controls

91 lines (78 loc) · 4.69 KB
Raw

Agent notes

This repo is a reverse-engineered, pure-Python reader for Tableau .hyper files (hyperparse.py). There is no public spec for the on-disk format — everything here was derived by reading Tableau's own engine binary and cross-checking against the real Hyper API as an oracle. If you are an agent picking this up cold, read this file before touching hyperparse.py.

Orientation — four rules that save you from repeating past mistakes

  1. Use the symbol-rich binary, not the Linux one. The Linux hyperd is fully stripped (objdump -t returns an empty symbol table). The macOS arm64 hyperd (bundled in the tableauhyperapi wheel) ships tens of thousands of demangled hyper:: symbols, including the whole storage engine. Get it with:

    pip download tableauhyperapi --no-deps --only-binary=:all: \
    --platform macosx_13_0_arm64 -d ./wheels

    Disassemble a named function directly, no Ghidra project needed:

    xcrun llvm-objdump -d --disassemble-symbols=<mangled-name> hyperd

  2. You have a ground-truth oracle. pip install tableauhyperapi gives you the real engine. hyperparse.py --verify FILE cross-checks a parse against it; the probe*.py scripts build synthetic .hyper files and check the results against the same API.

  3. Differential testing beats staring at hexdumps — but reading the function beats both. When a byte-level guess resists a brute-force sweep, that's a signal to go disassemble the relevant function instead of trying more guesses.

  4. Correlation is not mechanism. Don't infer a field's meaning or a format rule from a size/byte correlation observed in a single file. Build a second file that varies only the suspected cause before trusting it. Every retracted claim in this project's history came from skipping that step — see the corrections log in the full notes (next section).

Where the deep notes live

The full session-by-session derivation — every field offset, every compression scheme, the complete corrections log of retracted claims, open questions, and the tooling reference — lives in REVERSE_ENGINEERING.md in this directory. That file is gitignored on purpose (it's a working research log, not user-facing docs), so it will only be present in a checkout that already has it locally; a fresh clone won't have it. If it's missing and you need the detailed evidence behind a field or encoding, regenerate the understanding using the same method: read the engine, use the API as an oracle, write a probe.

Non-obvious gotchas worth knowing before you change decoding logic

These are the highest-cost lessons from that log, condensed:

  • Nullability is a property of the block, not the column. A nullable column with no NULLs in it gets a non-null scheme. Every Tableau-written column is declared nullable, so this bites immediately.
  • A DELETE does not rewrite Data Blocks. Deleted rows stay physically present; a separate Relation_Metadata tombstone B-tree records which ones are gone. A reader that ignores it returns the right values with too many rows.
  • Relations over 131,072 rows split into multiple blocks per column. Reading only the first block silently truncates.
  • String/Bytea/JSON columns move to the "native" (wide) dictionary entry once any value hits 256+ bytes — the "narrow" family's length is a single byte. Assuming a type family from the column's declared type alone is wrong.
  • The domain-size (distinct count) offset depends on the value's byte width, not on a fixed distance from the dictionary.
  • Integers are signed. An unsigned read passes every test until a negative value shows up.
  • Coverage claims are proven with a probe per claim, not a bigger corpus. Real-world corpora tend to under-exercise edge cases (no long strings, no negative numbers, no relation past the block cap) — a corpus that "cannot fail" isn't evidence.

Working in this repo

  • No third-party dependencies are required to run hyperparse.py itself — LZ4 block decompression and CRC-32C are implemented inline. Only the probe*.py scripts and checkrows.py need tableauhyperapi (used purely as a test oracle, never as a runtime dependency of the reader).
  • When you change decoding logic, re-run --verify against real files and, if tableauhyperapi is available, the relevant probe*.py script before calling it done. A change that only passes on sample_superstore.hyper is not verified — see the gotchas above for why.
  • tableau_hyper.ksy is a Kaitai Struct definition kept in sync with hyperparse.py; kscheck.py diffs the two over the corpus and should pass after any format change that touches both.