Add doc-test workflow for introduction.md examples#192
Merged
Conversation
A Python script (bin/test-doc-examples) extracts every fenced
factor block from concepts/*/introduction.md and
exercises/concept/*/.docs/introduction.md, runs each file's
blocks back-to-back inside the test-runner image with a single
docker run, and checks that the `! => ...` lines appear as a
subsequence of Factor's actual stdout. A GitHub Actions
workflow (.github/workflows/doc-test.yml) wires it into CI.
Initial baseline against main had 26 failing blocks across 14
files — real doc bugs of the same shape as the recent `7/2` /
`16/3` fixes, plus a handful of undefined-word typos and one
swapped-arg change-nth. This commit fixes all of them; the
script now reports 149/149 blocks pass across 48 files.
Notable doc fixes:
- concepts/tuples: `point new` actually prints `T{ point }`,
not `T{ point { x 0 } { y 0 } }` (Factor pprint omits slots
at their initial values).
- concepts/strings: `.s` output was rendered as `"foo" / "bar"`
but Factor prints stack items one per line; the surround
example had 4 strings and a TODO note (`"... err, surround
takes 3"`) and is now a clean 3-arg example.
- concepts/quotations & high-school-sweetheart: `{ 1 [ 2 + ]
[ 3 * ] } concat` doesn't type-check (concat needs sequence
elements) — wrap the leading literal in a quotation.
- pursers-pantry: hashtable iteration order isn't tied to
insertion order in Factor; drop the brittle `! => H{ ... }`
assertion on a 2-key literal and note the unordered nature.
Block 2 had set-at args swapped (printed `H{ { 5 "coal" } }`
instead of `H{ { "coal" 5 } }`).
- mosaic-mischief: change-nth args were ordered as `quot i
pick`, which mangles the stack; reorder to `i over quot`.
- lighthouse-logbook: blocks referenced an undefined `my-log`;
add a `: my-log` definition.
- lasagna-luminary, quayside-crew, concepts/errors,
factory-failsafe, high-school-sweetheart block 0,
concepts/streams block 0: marked `! DOCTEST: SKIP` for
illustrative blocks that re-define a word, throw outside a
recover, use `:>` outside `[let`, or reference a
yet-to-be-defined helper.
- secrets: multi-line parenthetical commentary inside an
`! => -12 (... )` comment was being parsed as part of the
expected output; lift the explanation into prose.
- streams block 1: `stream-contents` needs the stream on the
stack, but inside `with-input-stream` the stream is bound
to the dynamic `input-stream` variable; use `read-contents`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The kind of bug we just fixed in #190 (
! => 16/3where Factor actually prints5+1/3) is exactly what this catches automatically. Initial baseline against main found 26 such bugs across 14 files — wrong printed outputs, undefined-word typos, swapped `change-nth` args, etc.Notable doc fixes
Script details
Test plan