fix(docx): write a paragraph the page reads as markdown as the page sets it

[DemchaAV]

Why

A session reads markdown unless it is told not to (markdown(false)). The page then reads a paragraph of plain text that holds *, _ or a backtick through the engine's markdown parser. Text its marks style is set bold or italic, a heading line bold (the first three levels larger), and the marks are dropped. The DOCX export wrote the text as authored, so Word showed **bold** with its asterisks and none of the bold. The report named it, but a style Word can hold stayed unwritten.

TimelineMinimal shows it in the corpus. Its open-source project line reads … billing systems. *(Open source)*. The page sets the line in regular Lato with (Open source) in italic. Word showed the whole line bold, with the asterisks.

What changed

• DocxMarkdown (new, package-private) reads a paragraph's text as the page does:

  • line by line, after the same control-character pass;
  • each line through the engine's own MarkDownParser;
  • a line that opens with -, * or + and a space keeps that marker, as ParagraphWrapping.tokenizeMarkdownLine does;
  • each piece takes the face the parser sets and, for a heading line, the multiple of the size;
  • a heading keeps the paragraph's tracking in points, as the page does.

• DocxMarkdown.laidOutIn decides whether the pieces are written. It compares them with the lines the page laid the paragraph out in:

  • the non-whitespace letters must match, less a prefix that leads the first line;
  • each letter must have the same face, family, colour and tracking;
  • sizes must match to a hundredth of a point. The exception is an auto-sized paragraph, which the page fits to a size its style does not hold: there, sizes must stand in the same proportion, and tracking is not compared;
  • pieces with no letter at all never pass.

  A session with markdown off lays the marks out, so the check fails and the text is written as it stands. If the mirror of the core parse path drifts in letters, faces, families, colours, sizes or tracking, the check fails too: the paragraph is written as authored, and named where the page dropped marks. White space is not compared, because the page drops it where it breaks a line.

• markdownPieces keeps text the parser changes nothing of — every mark kept, every piece in the paragraph's own style, snake_case in a regular paragraph — written as it stands, white space and tabs included.

• writeParagraphRuns writes a plain-text paragraph as one run per piece where markdownPieces returns pieces:

  • the pieces of a linked paragraph go through runAfter, so they stay in one w:hyperlink;
  • the paragraph mark takes the last piece's style, as it takes the last run's;
  • markdownWritten records what was written, so the note leaves out what is written (markdownLost) and Word's outline lists a heading by the text written (outlineTextOf). Every path writes a paragraph before it makes its note. Recording what was written, rather than recomputing it in the report, keeps a path that writes as authored from being called written.

• Every path that writes a paragraph goes through this:

  • body, table cell, text over the flow and the line an overlay's two sides share all call writeParagraphRuns;
  • a page zone's line passes its parts' zone lines (zoneLinesOf, now shared with reportZoneLine);
  • a badge counts its initials from the pieces (badgePieces). **JR** is now a badge of two bold letters. *J*R, in two faces, is written in the flow, as initials in two runs' faces are, and so is a heading.

• markdownHeadingCut names a markdown heading written taller than the line the page sets it in. The page sets a heading line as tall as the paragraph's own and draws its letters past it; written in that exact line, Word cuts their tops on screen. The check compares the heading's own line height with the page's line. An auto-sized paragraph's heading, written at a multiple of its style's size, often fits the taller line the page fits the text to, and is then not named.

• DocxLayoutMetrics.matchComposedText pairs a paragraph composed in a table cell with a fragment by its text as the page reads it (marks dropped). It accepts only a fragment whose lines set its pieces (setsThePieces). A plain paragraph of the same text later in the table takes the first fragment of that text in the first pass, which may be this one's own. The fragment left over is then in another face or size, and the markdown paragraph takes no line rather than be cut by it. Before this PR, the page dropped the marks, so the authored text never matched and such a paragraph got no lines at all.

• DocxFontTable.collectFonts adds the faces of the pieces of the paragraphs it reads (outside table cells and page zones, as before). The table is written before any paragraph, so it reads the faces off the text. A session with markdown off therefore ships a face it does not use.

• parserDropsAMark reads through DocxMarkdown.read, line by line and with the list-marker rule, rather than parsing the whole text at once. With no layout, * a_b is no longer named.

• Still written as authored, and named:

  • a paragraph whose lines are not read: with no layout, composed in a cell whose text no line of its table carries that way, or a page zone part the layout shows none of;
  • one the page sets in other letters than its text, such as Arabic, which the page shapes before it parses;
  • text the parser reads into nothing — a lone * (an empty list item), *** (a rule), a line set four spaces in (a code block it keeps no text of) — which the page sets as nothing. These went unnamed before. Written as nothing, the paragraph would be blank, and the table-row code (takeFromTheFoot) treats a blank paragraph as a cell taking no room; a fixture in DocxLinePairTest that marks a cell with a lone * shows it;
  • list items, unchanged in this PR.

The faces are the page's

The page's markdown parser sets every piece in a face of its own and drops the paragraph's own face. A bold paragraph whose text holds a mark is therefore drawn regular, with only the marked pieces bold or italic. The export follows the page. TimelineMinimal's project line is a bold paragraph, drawn regular next to its bold neighbours, and Word now draws it regular too. This is engine behaviour, outside this PR. thePiecesStandInTheFacesThePageSetsThemIn pins the page's face, so a change in the engine fails that test, and the export, which compares faces with the page's lines, follows.

Verification

• ./mvnw -B -ntp install -pl :graph-compose-render-docx → BUILD SUCCESS, 1202 tests, 0 failures, 1 skipped.
• New DocxMarkdownTest (6) covers:
  • pieces for emphasis, code, links, headings, list-marker lines and blank lines;
  • the parser's face winning over the paragraph's;
  • heading tracking in points;
  • laidOutIn true for wrapped lines, auto-sized proportions and a leading prefix;
  • laidOutIn false for laid-out marks, marks alone, another face, family, colour, tracking or letter, a proportional size where the page fits none, sizes out of proportion, a missing letter, a prefix of other letters, no lines, and a non-text span.
• New DocxSessionMarkdownTest (21) covers:
  • written runs with their faces;
  • a heading at twice the size, with its note, the mark's size and line breaks;
  • a direction mark;
  • a right-to-left paragraph;
  • a paragraph broken across pages;
  • text the parser reads into nothing (***, *, a line four spaces in), written and named;
  • markdown off, and text the parser keeps whole (with a tab) in either session;
  • a session-level check against the engine: an ordered-list line, a quote, escapes, nested emphasis, a code span holding _, a link, an entity, double spaces, and a tracked heading — each written as the page sets it;
  • an auto-sized paragraph, written in pieces, and its heading not named where it fits;
  • the page's face for a bold paragraph, pinned;
  • one link;
  • outline text;
  • a footer zone, with a heading in it named;
  • a composed cell, with the bold run;
  • a cell keeping its own line beside a markdown cell whose prefix hides its text;
  • a markdown cell taking no line of another face or size;
  • badges: one face, **JR**, *J*R, and a heading moved to the flow;
  • the two sides of a line pair, with the outline;
  • text over the flow, its bold run;
  • italic and bold faces in the font table;
  • Arabic, still named.
• DocxMarkdownReportTest: paragraphs, cells and zones are written and not named, a heading is named, * a_b with no layout is not, and the no-layout and list notes are otherwise unchanged.
• 38 sabotages, each one reverting a single decision in the change, are each caught by a test. They cover:
  • each check in laidOutIn, tracking included;
  • every write path;
  • the notes: marks, nothing set, and the heading cut by line height and in a zone;
  • the outline text and the link grouping;
  • the cell pairing, its order, and the check that a cell's line sets its pieces;
  • the font table;
  • the badge count and its two-face and heading rules;
  • pieces that change nothing;
  • line-by-line reading.
• Core doc guards: 166/0. qa doc and DOCX guards: 50/0.
• DOCX fidelity corpus: 61 of 62 documents are byte-identical. TimelineMinimal changes: its project line is written in regular Lato with (Open source) in Lato Italic, and the file ships the italic face (365 KB larger). In Word 16.0 the line renders as the engine PDF draws it, where before it was bold with asterisks. Report notes: 954, from 955 (that line's note).

Known limits

• List items the page reads as markdown are still written as authored and named; they come next.
• An auto-sized paragraph is still written at its style's size, a heading's pieces at its multiple; that comes after lists.
• A heading in a page zone is named, as in the body, rather than given a taller zone line.
• The font table reads no paragraph composed in a table cell or set in a page zone, for any face, as before this change.
• A plain paragraph in a table cell may still take the line of a markdown paragraph of the same text before it, as before this change; that line is then taller or shorter than its own.

Lane: render-docx (DOCX semantic backend) — no public API change.