Skip to content

feat(web): add bounded transcript content search - #387

Open
testikun wants to merge 1 commit into
openpi-dev:mainfrom
testikun:codex/issue-349-transcript-search
Open

feat(web): add bounded transcript content search#387
testikun wants to merge 1 commit into
openpi-dev:mainfrom
testikun:codex/issue-349-transcript-search

Conversation

@testikun

@testikun testikun commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Problem

Issue #349 requires transcript-aware server search, but the current Web snapshot exposes only a bounded title/cwd/first-message projection. The open metadata-search PRs do not provide a safe mechanism for scanning Session JSONL content or tool evidence.

Relates to #349. This PR implements the transcript-scanner slice only; it does not duplicate PR #362's endpoint or PR #376's pagination work.

Value

  • Gives the Web host a reusable server-side primitive for finding prior user/assistant text and bounded tool evidence.
  • Keeps Pi Session files authoritative while making partial, malformed, changed, unavailable, and budget-limited outcomes observable.
  • Preserves exact Session/message/turn/workspace/source/timestamp provenance without retaining an unbounded index.

Approach

  • Add searchWebTranscripts() over a caller-authorized Session catalog; the scanner never discovers or widens filesystem authority.
  • Enforce hard caps for catalog files, total bytes, per-file bytes, line bytes, retained results, snippet bytes, and wall time; support AbortSignal cancellation.
  • Require regular non-symlink files within canonical allowed roots and exact workspace membership, then validate the Session header identity before publishing matches.
  • Scan JSONL incrementally in fixed-size chunks, retain only a bounded newest-result set, propagate turn identity across intervening Pi entries, and re-check file identity after scanning.
  • Search user/assistant text plus tool call/result evidence, while excluding private result details. Return control-safe, bidi-safe, UTF-8-bounded plain-text snippets with explicit truncation evidence.
  • Fail soft per file for malformed, changing, deleted, oversized, and unauthorized inputs, with exact partialReasons and accounting.

The existing lightweight metadata filtering remains separate. Endpoint composition, pagination/cursors, and derived index invalidation remain follow-up slices of #349.

Validation

  • node --test --experimental-strip-types tests/web/transcript-search.test.ts — 8 passed
  • bun run check — passed
  • bun run test — Node: 1249 passed, 0 failed, 1 platform skip; Vitest: 30/30 passed
  • Test coverage includes a real Pi-written Session file, user/assistant/tool evidence, provenance, Unicode snippets, control/bidi stripping, malformed JSONL, archive filtering, root/workspace/symlink rejection, header mismatch, every budget, timeout, and cancellation.
  • Manual UI validation: not applicable; this PR adds no UI behavior.

Impact

  • User-visible behavior: None until a Web endpoint/UI composes the scanner.
  • Model-visible context/tools: None.
  • Runtime/lifecycle: Adds a read-only, cancellable, bounded Session-file scanner; no second Session store or index.
  • Persisted config/data: None.
  • Compatibility/risk: Additive module only. The caller must supply the authorized Session catalog and roots; the scanner fails closed on provenance or path mismatches.

@testikun

testikun commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

CI status note: Node 22/24 completed this PR's repository checks and failed only at the final fresh-install Smoke-test packed standalone Web CLI step, where the child exits before the test can read its URL. The same failure now occurs on unrelated PRs #385, #386, and #387.

I reproduced the packed install independently: npm now admits Pi 0.85.0 through OpenPI's published peerDependencies: "*"; the CLI then exits with Cannot find module '@earendil-works/pi-server'. The checkout/lock-based bun run check and full test suite remain green. I recorded the shared packaging regression and suggested peer-range follow-up in #328: #328 (comment)

No dependency workaround is mixed into this feature PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant