feat(detect): classify Typst (.typ) as documents - #2921
Conversation
.typ was in no extension set, so classify_file() returned None and a repo that authors its report/spec chapters in Typst lost every one of them - Markdown and the generated PDF were indexed while the sources that produced them were not. Adding .typ to DOC_EXTENSIONS also carries watch, since _WATCHED_EXTENSIONS is derived from it; there is a test pinning that coupling because it is easy to miss. .typ also joins _SPLITTABLE_TEXT_SUFFIXES. That module's precondition is a plain-text document whose _file_to_text is a straight read_text, and .typ meets both (_file_to_text only special-cases .pdf). Without it a brand-new document type would land on the wrong side of _FILE_CHAR_CAP and every oversized Typst chapter would be silently truncated instead of sliced (Graphify-Labs#2900). Typst heads sections with = rather than #, so the heading boundary never fires and the blank-line boundary carries it - the same fallback .txt already uses. No AST extractor: Typst is not Markdown, and routing it to extract_markdown would manufacture wrong structure. .typ goes through the semantic document pass like .txt, which is the scope the issue asks for. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
Adds .typ (Typst) as a recognized document type by extending DOC_EXTENSIONS in detect.py and _SPLITTABLE_TEXT_SUFFIXES in file_slice.py, so Typst files classify as documents, get watched, and slice at paragraph boundaries when oversized. Covers the new behavior with detect, watch, and file-slice tests.
No blocking issues surfaced.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1656 functions depend on the 535 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 474 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
detect()— 109 callers, 15 callees - new:
save_manifest()— 39 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
dispatch_command()— 2 callers, 119 callees - …and 27 more — each is listed as a finding
Verification — 1656 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 941 function(s) in the blast radius were not formally verified this run
· 35 more finding(s) on lines outside this diff (see the check run).
1 participant
Fixes #2826.
The problem
.typwas in no extension set, so a repository that authors its report/spec/datasheetchapters in Typst lost all of them:
The reporter's case is the common one — Markdown and the generated PDF were picked up while
the Typst sources that produced them were invisible.
The change
.typjoinsDOC_EXTENSIONS. That is the one-line fix the issue proposes, and it alsocarries
watchfor free:_WATCHED_EXTENSIONSis derived asCODE_EXTENSIONS | DOC_EXTENSIONS | PAPER_EXTENSIONS | IMAGE_EXTENSIONS, so editing a.typchapter now triggers an incremental rebuild. There is a test pinning that, since the coupling
is easy to miss.
It also joins
_SPLITTABLE_TEXT_SUFFIXESinfile_slice.py. That module's stated preconditionis "plain-text document types ... where
_file_to_textis a straightread_text, so a charrange matches the bytes the model is shown" —
.typsatisfies both (_file_to_textonlyspecial-cases
.pdf). Without it, a new document type would be born on the wrong side of_FILE_CHAR_CAPand every Typst chapter over the cap would be silently truncated rather thansliced, which is exactly the complaint in #2900. Typst heads sections with
=/==rather than#, so the"\n#"boundary never fires, but the blank-line boundary carries it — the samefallback
.txtalready relies on.No AST extractor is added: Typst is not Markdown, and mapping it onto
extract_markdownwouldmanufacture wrong structure.
.typgoes through the semantic document pass like.txt, whichis the scope the issue asks for ("a richer follow-up could add Typst-aware text cleanup, but
simple document classification would already make Typst-authored repos much more usable").
Tests
Four, all failing on
v8before the change:test_detect_surfaces_typst_chapters_as_documentsruns the issue's repro throughdetect()rather than stopping at
classify_file, because "appears in the document set" is the actualpromise that was broken. The slicing test asserts the slices reconstruct the file byte for
byte, so nothing is dropped.
Verification
Windows 11 / Python 3.12.10,
v8@b2cd362.Full suite: no new failures — the failure set is identical to
origin/v8's (50 pre-existing,all optional-dependency, CI-history, or POSIX-only fixtures; see #2919).