An oversized PDF loses everything past 20k characters: slicing reads the container's bytes, so PDFs are excluded from it entirely

[abhay-codes07]

Summary

expand_oversized_files slices oversized documents so the whole file still reaches the model, but it measures candidates with path.read_text() and read_slice_text slices the same way — while llm._file_to_text routes a PDF through extract_pdf_text. Slicing a PDF would therefore index the container's bytes rather than its text, so PDFs are excluded from slicing entirely and simply lose everything past _FILE_CHAR_CAP (20,000 characters).

This is the half #2900 called out and left alone. Filing it separately because it is independently reproducible and papers are the longest documents anyone points graphify at.

Reproduce

A one-page PDF with a real text layer, FlateDecode as every real PDF is:

pdf bytes=3094   extracted chars=55690   cap=20000
units after slicing: 1
chars reaching the LLM: 20000   LOST: 35690

64% of the document never reaches the semantic pass. Nothing in it can become a node, and nothing reports the loss: there is no warning and no _partial_files marker (that machinery tracks truncated responses, not truncated inputs).

Why a PDF hides this better than a text file

A compressed PDF gives no hint of its text length — 3,094 bytes on disk against 55,690 characters of text. Neither the file listing nor the corpus word count makes the gap visible, and the graph that comes out looks complete: a paper whose second half was cut is indistinguishable from one whose second half had nothing extractable in it.

Suggested fix

One reader that returns the string the prompt will carry, used by both the boundary pass and the slice reader, so the offsets a FileSlice holds always index the same string the model is shown:

def unit_source_text(path: Path) -> str:
    if path.suffix.lower() not in _CONVERTED_TEXT_SUFFIXES:
        return path.read_text(encoding="utf-8", errors="replace")
    return extract_pdf_text(path)          # memoised

Then is_splittable_text can answer the question slicing actually asks — "can this be addressed by character offset?" — rather than "is this one of five suffixes". Images and code stay out; PDFs come in.

Worth memoising on (path, size, mtime): slicing asks for the same file once to measure it and then once per slice, and extraction is the expensive part. Keying on size and mtime keeps a paper replaced mid-run from being sliced against stale text.

extract_pdf_text lives in detect, and llm imports file_slice (not the reverse), so a lazy import from file_slice into detect avoids a cycle.

Related

• Oversized .qmd/.html/.yaml/.skill documents are silently truncated at 20k chars before the semantic pass — only 5 of 10 document types are sliceable #2900 / fix(llm): slice every text document type, not just five of them (#2900) #2902 — the same drift for .qmd, .html, .yaml, .skill; that PR fixed the plain-text half and explicitly deferred this one.
• PDF token estimates are computed from the compressed container, not the extracted text — chunks overfill by ~3.5x and get bisected #2903 / fix(llm): estimate PDF tokens from the extracted text, not the container (#2903) #2904 — the same "raw file vs converted text" confusion in _estimate_file_tokens. Independent: that one bites a PDF under the cap, this one bites a PDF over it.

I have this working with tests and will open a PR shortly.

Environment

graphify	v8 @ b14b52e (0.9.47)
Python	3.12, Windows 11