Skip to content

Add doc-test workflow for introduction.md examples#192

Merged
keiravillekode merged 1 commit into
exercism:mainfrom
keiravillekode:doc-test
May 11, 2026
Merged

Add doc-test workflow for introduction.md examples#192
keiravillekode merged 1 commit into
exercism:mainfrom
keiravillekode:doc-test

Conversation

@keiravillekode
Copy link
Copy Markdown
Contributor

Summary

  • New script `bin/test-doc-examples` extracts every ```factor``` block from `concepts//introduction.md` and `exercises/concept//.docs/introduction.md`, runs each file's blocks back-to-back inside the `exercism/factor-test-runner` image with a single `docker run`, and checks that each `! => ...` line appears as a subsequence of Factor's actual stdout.
  • New workflow `.github/workflows/doc-test.yml` wires it into CI on the same shape as `test.yml` (`ubuntu-24.04`, pull the image, run the script).
  • Fixes the 26 doc bugs the script surfaced on `main` so the workflow lands green. Final score: 149/149 blocks pass across 48 files.

Why

The kind of bug we just fixed in #190 (! => 16/3 where Factor actually prints 5+1/3) is exactly what this catches automatically. Initial baseline against main found 26 such bugs across 14 files — wrong printed outputs, undefined-word typos, swapped `change-nth` args, etc.

Notable doc fixes

  • `concepts/tuples`: `point new` actually prints `T{ point }`, not `T{ point { x 0 } { y 0 } }` (Factor pprint omits slots at their initial values).
  • `concepts/strings`: `.s` output was rendered as `"foo" / "bar"` on one line, but Factor prints stack items one per line; the `surround` example had 4 strings and a TODO note (`"... err, surround takes 3"`).
  • `concepts/quotations` & `high-school-sweetheart`: `{ 1 [ 2 + ] [ 3 * ] } concat` doesn't type-check — `concat` needs sequence elements.
  • `pursers-pantry`: hashtable iteration order isn't tied to insertion order; drop the brittle assertion. Block 2 had `set-at` args swapped (printed `H{ { 5 "coal" } }` instead of `H{ { "coal" 5 } }`).
  • `mosaic-mischief`: `change-nth` args were `[ 1 + ] 0 pick` which mangles the stack; reorder to `0 over [ 1 + ]`.
  • `lighthouse-logbook`: blocks referenced an undefined `my-log`; add a `: my-log` definition.
  • `secrets`: multi-line parenthetical commentary inside an `! => -12 (... )` comment was being parsed as part of the expected output; lifted into prose.
  • `streams` block 1: `stream-contents` needs the stream on the stack, but inside `with-input-stream` it's bound to the dynamic `input-stream` variable; use `read-contents`.
  • Marked `! DOCTEST: SKIP` on a few intentionally-untestable blocks (re-definitions for comparison, illustrative `throw` outside a `recover`, `:>` outside `[let`, hypothetical helpers).

Script details

  • Each fenced ```factor``` block in a file gets composed into one Factor program, with a sentinel `print` between blocks so we can localise failures.
  • `! => result` lines are extracted as expected output. Trailing prose (` (a proper ratio)` etc.) is stripped before comparison.
  • Subsequence matching tolerates demo lines that print without an explicit `! =>` — extra actual output is fine, but expected lines must appear in order.
  • If no block in a file declares `USING:` we prepend a generous default; otherwise we trust the doc (mixing the default with a per-block `USING:` provokes ambiguous-word errors — `ascii` and `unicode` both define `blank?`/`LETTER?`).
  • A block can opt out with `! DOCTEST: SKIP`.

Test plan

  • CI on this PR runs `Doc-test` to green.
  • CI on this PR continues to run `Track Exercises on Runner` to green.

A Python script (bin/test-doc-examples) extracts every fenced
factor block from concepts/*/introduction.md and
exercises/concept/*/.docs/introduction.md, runs each file's
blocks back-to-back inside the test-runner image with a single
docker run, and checks that the `! => ...` lines appear as a
subsequence of Factor's actual stdout. A GitHub Actions
workflow (.github/workflows/doc-test.yml) wires it into CI.

Initial baseline against main had 26 failing blocks across 14
files — real doc bugs of the same shape as the recent `7/2` /
`16/3` fixes, plus a handful of undefined-word typos and one
swapped-arg change-nth. This commit fixes all of them; the
script now reports 149/149 blocks pass across 48 files.

Notable doc fixes:
- concepts/tuples: `point new` actually prints `T{ point }`,
  not `T{ point { x 0 } { y 0 } }` (Factor pprint omits slots
  at their initial values).
- concepts/strings: `.s` output was rendered as `"foo" / "bar"`
  but Factor prints stack items one per line; the surround
  example had 4 strings and a TODO note (`"... err, surround
  takes 3"`) and is now a clean 3-arg example.
- concepts/quotations & high-school-sweetheart: `{ 1 [ 2 + ]
  [ 3 * ] } concat` doesn't type-check (concat needs sequence
  elements) — wrap the leading literal in a quotation.
- pursers-pantry: hashtable iteration order isn't tied to
  insertion order in Factor; drop the brittle `! => H{ ... }`
  assertion on a 2-key literal and note the unordered nature.
  Block 2 had set-at args swapped (printed `H{ { 5 "coal" } }`
  instead of `H{ { "coal" 5 } }`).
- mosaic-mischief: change-nth args were ordered as `quot i
  pick`, which mangles the stack; reorder to `i over quot`.
- lighthouse-logbook: blocks referenced an undefined `my-log`;
  add a `: my-log` definition.
- lasagna-luminary, quayside-crew, concepts/errors,
  factory-failsafe, high-school-sweetheart block 0,
  concepts/streams block 0: marked `! DOCTEST: SKIP` for
  illustrative blocks that re-define a word, throw outside a
  recover, use `:>` outside `[let`, or reference a
  yet-to-be-defined helper.
- secrets: multi-line parenthetical commentary inside an
  `! => -12 (... )` comment was being parsed as part of the
  expected output; lift the explanation into prose.
- streams block 1: `stream-contents` needs the stream on the
  stack, but inside `with-input-stream` the stream is bound
  to the dynamic `input-stream` variable; use `read-contents`.
@keiravillekode keiravillekode merged commit 2716888 into exercism:main May 11, 2026
8 checks passed
@keiravillekode keiravillekode deleted the doc-test branch May 11, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant