pdfToMarkdown discards page boundaries and heading structure

[fcbwilliams]

Summary

pdfToMarkdown returns the document's text essentially complete, but with almost no structure: no
headings, no reliable paragraph boundaries, and no page markers. For text-extraction consumers —
search indexing, RAG chunking, LLM context — structure is most of the value, because it is what
chunk boundaries are derived from.

Observed

Round-tripping a document with one # title, two ## sections, 60 paragraphs and a 2-column
GFM table through markdownToPdf → pdfToMarkdown (documents.js@1.100.1):

heading lines in output          →  []      (the title and both sections became **bold** runs)
60 source paragraphs             →  8 blank-line-separated blocks
pipe-table lines                 →  0       (the table became tab-separated inline text)
every string of source content   →  present (no text lost)

Sample of the output, showing a heading merged into the body text that follows it:

**Quarterly Technical Report**

**Volume 2**

**Part 1 Scope **This is body paragraph 0 of ordinary prose content that follows the heading ...

Text fidelity is genuinely good. Structural fidelity is what is missing.

Why this is inherent, and where it isn't

Most of this is honest: PDF has no semantic headings, so reconstructing them from geometry is
inference, and the README's Fidelity section already rates pdf → markdown as severely lossy. #565's
distinction is the useful frame — the content-pivot markdown conversions (docxToMarkdown etc.)
have ContentParagraph.styleId to infer headings from, whereas PDF input arrives through layout
reconstruction, where no such field exists.

But one part is not inference at all: LayoutDocument knows exactly how many pages there are, and
where each begins. That information is present, exact, and currently discarded on the way to
Markdown.

Asks, in order of confidence

1. Emit page boundary markers. A per-page marker in pdfToMarkdown output (### Page 3, an HTML
   comment, or an option to choose) is exact rather than heuristic, costs nothing, and gives
   downstream consumers a reliable break to chunk on. This is the one I would most like.
2. Infer heading level from font size during reconstruction. Genuinely heuristic, and the same
   category of heuristic reconstructWordprocessing already applies for paragraph boundaries from
   baseline spacing — relative font size against the document's modal body size is a well-worn
   signal. Emitting ## instead of **bold** for a run 1.5× body size would help every consumer.
3. Recover tables into GFM when a gridline lattice was detected. The README says
   PDF → docx/odt already recovers tables "from a real gridline lattice", so where that succeeds, the
   structure exists and could reach Markdown as a pipe table rather than tab-separated text.

(1) is small and unambiguous. (2) and (3) are worth discussing rather than assuming — happy to be
told they are out of scope for what pdfToMarkdown is meant to be.

[fasuizu-br]

Affiliation: I am writing on behalf of BRAINIALL, which operates a metered hosted text-layer PDF-to-Markdown route; it does not provide OCR or claim structural fidelity. I will not open a PR, request credentials, or use a private or non-rights-cleared document. Since #584 separates exact page boundaries from heuristic heading and table recovery, would one maintainer-selected rights-cleared synthetic two-page fixture be useful as a single external baseline recording the fixture hash, page-range request, returned Markdown, page-boundary, heading and table observations, response MIME, elapsed time, and observed charge before you decide implementation scope? This would not claim documents.js compatibility, exact page markers, heading or table recovery, layout fidelity, OCR, or production quality. If you opt in, I will use only the fixture and gates you select; silence for 7 days ends the route with no follow-up.