Add TokenScope Open List build (Word selection rewrite with a live token counter)

[AbhinayYendoti]

What I built

TokenScope — a Word add-in that rewrites the paragraph you selected, then shows what that surgical edit actually cost next to what regenerating the whole document would have cost.

Built for the Open Task List card "Word selection rewrite with a live token counter" (Band S1 — surfaces: chat, API).

Path: extensions/AbhinayYendoti/tokenscope/

Problem

An AI document editor that regenerates the whole document on every turn pays for the whole document on every turn. The cost of a one-paragraph change scales with the size of the file it happens to live in, which is backwards — the edit did not get bigger, the document did.

Surgical editing breaks that coupling. The claim is easy to make and easy to hand-wave, so TokenScope makes it falsifiable: it runs both operations against the live API, on the same document, on the same day, and reports what each one actually did.

How it works

Word selection (Office.js)  →  rewrite instruction  →  SuperDocs chat/async
      →  proposed change (not yet applied)  →  token measurement
      →  whole-document comparison  →  review  →  Apply

The selection is delimited in the message rather than described, so the model matches an exact string instead of interpreting "the second paragraph". Every figure on screen carries a badge saying whether it was Measured, Tokenized or Estimated.

SuperDocs integration

Live REST API throughout — no mock, no offline fallback. Without a key the app starts, says the key is missing, and refuses to rewrite rather than show invented numbers.

Surface	Use
POST /v1/chat/async	Both operations. Chosen over synchronous /v1/chat, which returns before a job record exists — and the job record is the only place a token count appears.
GET /v1/jobs/{job_id}	Polling (compact=true while running, one full read once settled), and the source of the measured token count.
POST /v1/chat/{session_id}/approve	Applying one named change, only when the user presses Apply.
GET /v1/agents/whoami	Tier and remaining quota in the pane header.
Human-in-the-loop	approval_mode: "ask_every_time" on every request; approve_all is never sent.

The token counter is measured, not asserted. It reads metadata.cumulative_tokens off the job record — SuperDocs' own figure, never computed here. The whole-document side is not modelled either: it is the same instruction run as a genuine regeneration in a throwaway session on a copy of the document.

That field turned out to be unstable across runs and is sometimes absent entirely, so it is nullable and renders as "Not reported" rather than zero. A second measurement sits beside it — the tokens of text each job actually returned, counted from its own pending_changes — which is available on every job at every size.

Scope is verified. SuperDocs mints its own chunk ids, so returned changes are mapped back to the selection on their original text. A change that cannot be mapped is out of scope: shown to the user, never offered for apply.

How to run

cd extensions/AbhinayYendoti/tokenscope
npm install
cp .env.example .env.local     # add SUPERDOCS_API_KEY
npm run dev                    # http://localhost:5173

Everything runs from the project folder; no command needs the repository root. Word is optional — without it the right-hand column is a live document surface using the browser's own Selection API, and every SuperDocs call is identical. Word instructions (HTTPS cert, sideloading) are in the project README.

Testing

Run from extensions/AbhinayYendoti/tokenscope/ after a clean npm ci:

• npm test — 93 tests passing, no API key required
• npm run typecheck — clean (strict, exactOptionalPropertyTypes)
• npm run lint — clean
• npm run build — clean
• npx office-addin-manifest validate manifest.xml — valid

The client tests stub fetch rather than the client, so the real request is built, the real zod schemas parse the response, and the real error mapping runs; fixtures are trimmed copies of bodies the live API actually returned.

Manual end-to-end against the live API, from this folder: whoami returned tier and quota; a real rewrite completed in 9.4s producing one change, in scope, zero out of scope, with a measured cost of 86,702 tokens; the apply path was exercised separately in the browser.

npm run bench regenerates the benchmark. Results in bench/RESULTS.md are generated, never typed by hand, and every row carries its job ids and timestamp.

Document	Sections changed	Written, surgical	Written, regenerated	Saved
3 pages	1 vs 20	26 tokens	1,124 tokens	97.69%
10 pages	1 vs 61	18 tokens	4,589 tokens	99.61%
50 pages	1 vs 300	37 tokens	19,822 tokens	99.81%
100 pages	1 vs 599	32 tokens	43,344 tokens	99.93%
300 pages	1 vs —	30 tokens	never finished	see below

The surgical column does not move. That is the entire argument.

Limitations

• metadata.cumulative_tokens is not fully dependable. Not a running total despite the name; across runs the same 3-page rewrite reported 181,978 once and ~90,500 three times, and one 300-page run finished carrying no count at all. That is why the second measurement exists.
• The 300-page whole-document regeneration never completed. Three attempts reached 99% and stopped; the platform's own watchdog failed the last as wedged. The surgical edit on the same 150,000-word document completed every time in 217–340s. Recorded as a run that did not complete rather than converted into a percentage — a limit observed on a free-tier account at core, not a claim about the platform.
• o200k_base is probably not SuperDocs' tokenizer. Local counts measure text volume reproducibly and are only ever compared against each other, never presented as billing.
• The Word host is not covered by automated tests. Office.js needs a real Word process. The logic it delegates to is tested through the demo host and the shared layer; the Office.js calls in src/host/word.ts are unverified.
• The benchmark corpus is generated, not sampled — deterministic prose, 500 words/page, one instruction, one model tier.
• No screenshots or hosted demo; the standalone browser mode is the demo.

AI disclosure

AI coding assistance (Claude) was used throughout, as CONTRIBUTING invites. The measurement design, the decision to stop trusting the provider's counter after the benchmark contradicted it, and every number reported here came out of running the live API and reading what came back.

Candidate

Abhinay Yendoti