[oscarlevin]

Just like we do for the main image types, it would be helpful to let the CLI cache individual webwork representation files (and any other files, like images needed for the problems). This is possible now that we don't build the one giant representations file.

This is set up the same way as the image caching. Split out the parts of the webwork generation script that take the result of extract_pg.xsl and send them to the server (or process locally) to get the representation and other files. Core calls this (individual_webwork_conversion) when webwork_to_xml has ext_converter = None. But the CLI can set ext_converter to its own individual webwork generating function, that first checks the cache before calling core's individual_webwork_conversion if needed.

One tweak to xsl was required: we need to extract the pg into individual files to get the hash that the cli uses.

Claude tested before and after, both with the CLI and only pretext script, on a document with 500 webwork questions. Before and after with the pretext script took 114 and 113 seconds (so the extracting individual pg files didn't cost anything). It took 115 seconds for the CLI on first run (nothing cached), but then changing just one problem took only 5.6 seconds (5.2 seconds with nothing changed). This should make @sean-fitzpatrick very happy.

Also heads up for @Alex-Jordan. I don't think there is anything functionally different about the webwork code, except how it is organized.

[rbeezer]

Review coming from Claude.

• Needs a rebase anyway (my fault!) as some minor WW cleanup has gone in lately.
• A few little things need attention.
• Some (unrelated) improvements!
• Looks like something substantial to consider about caching.

[rbeezer]

Thanks, Oscar. Moving the conversion to one problem at a time looks right, and it reproduces master's output exactly. Notes below, the first one the most important.

Testing

I built the WeBWorK representations of examples/webwork/sample-chapter (76 problems, two of them copies) against the public server, rebased onto current master:

run	result	time
master	76 representation files, 38 images, 105 .pg files	34.1 s
this PR, through pretext/pretext	same	33.1 s
this PR, through a short script that passes an ext_converter	same	34.0 s

• Representation files and .pg files: byte-identical across all three runs.
• Images: eleven server-made LaTeX images differ, but they also differ between the two runs of this PR. The PDFs differ only in creation time and render identically; the PNGs are pixel-identical.
• Return value of individual_webwork_conversion(): the 76 calls returned 114 files, exactly the 114 written outside pg/. The representations file always comes first, and every name begins with its problem's identifier.
• Per-problem files: byte-identical across three separate extractions, so a hash of them is a stable cache key.
• Copies: their per-problem files carry copied-from-path naming the original's .pg file, and their representations match master's.
• Not run: static-processing="local". script/webwork/pg-ptx.pl needs the Perl Mojolicious modules, which the test machine lacks. With whitespace ignored, the changes in the local branch are the same origin[problem] → origin substitutions as the server branch, plus the list of written images.

1. The per-problem file does not hold everything a problem's rendering depends on

The new comment in extract-pg.xsl says "Everything that determines the exercise's representations is in that file", and PreTeXtBook/pretext-cli#1263 names each cache entry by a hash of that file. Some inputs are named in the file, but their content is not in it.

• Local rendering. pg-ptx.pl reads macro files from disk: author macro files in external/macros (listed by name in pg-macros/macro-file), and the document's own macro file (WWSC.pl for the sample chapter), which holds latexImagePreamble() built from docinfo/latex-image-preamble. To test, I changed grid = both to grid = none in the sample chapter's preamble and extracted again. All 76 per-problem files were byte-identical. With the CLI cache, the images in those problems keep the old preamble until the cache is cleared.
• Server rendering. Any of these can change without changing a per-problem file: a problem hosted on the server (@source), the course's macro files, or the server's version, which is written into every representations file.

With ext_converter set to None, every problem is rendered every time, as before, so the core conversion is unaffected. The comment should state what the file really guarantees. On the CLI side, there are two choices: fold the macro files into the hash, or tell authors when to clear the cache.

2. Two changes in behavior, both improvements

• A subtree build with a copy whose original is outside the subtree. Master looked up the original's path in a dictionary of the extracted problems. To test, I added <exercise label="copy-across-sections"><webwork copy="webwork-add-numbers" seed="7"/></exercise> to the end of the sample chapter's section-the-quadratic-formula, then ran -c webwork -r section-the-quadratic-formula. Master stops with KeyError: 'integer-addition'. This PR builds all four problems and gives the copy the original's .pg path. The CLI passes its xmlid through to webwork_to_xml() here, so this happens there too.
• No usable delimiter for a \verb expression. Master logged a warning and returned from webwork_to_xml(). That silently dropped every remaining problem and left the session and socket open. This PR stops with a PTX:ERROR. A return inside the new function would hand the CLI a string instead of a list of files, so some change was needed here, and an error is a good choice. The case needs every one of 87 printable characters in a single expression, so it is unlikely to be met.

3. Comment alignment in xsl/extract-pg.xsl

A comment of several lines closes every --> in the same column.

• Lines 233–234: the two lines added to the "1b" comment close at column 75. The four lines above them close at column 84.
• Lines 143–144: the new comment above ww-extraction-settings closes at columns 75 and 76.

4. Small duplication

• Origin checked twice. webwork_to_xml() now checks every problem's origin before converting, and individual_webwork_conversion() checks it again. The stylesheet's match pattern only produces generated or webwork2, so the first loop could go.
• Identifier computed twice. In extract-pg.xsl, the extraction and problem templates both compute the assembly identifier. The second could receive it as a parameter.

5. Commit subject

• Topic. The topic is Webwork:, which the history has never used; WeBWorK: has 197 commits.
• Length. The subject is 84 characters, over the 60 to 70 in AGENTS.md, and it describes the mechanism rather than the outcome. A suggestion: WeBWorK: build representations one problem at a time.
• Fine as it is. There is no body, which is right. One commit is right too, since the Python reads files the stylesheet writes. There is no stray whitespace.
• Optional. The moved lines carry three old typos ("wanrnings", "acctual", "inteentionally"). Every one of those lines is already in the diff.

Claude Opus 5.5, acting as a review assistant for Rob Beezer

[rbeezer]

At least two more WW related PRs, #3315 being one. I've not checked the effect on a rebase for thiose.

[oscarlevin]

Thanks, Rob (and Claude). Rebased onto current master (no conflicts with #3314 or #3315) and force-pushed as a single commit, WeBWorK: build representations individually. Taking the notes in order:

1. What the per-problem file guarantees. You're right, it names macro files without holding them. Two changes here:

• The comment in extract-pg.xsl now says what the file holds: everything the PreTeXt source says about the exercise, plus the settings it's rendered under. It also lists what it does not hold: macro files (an author's in the external directory, and the document's own, built from docinfo), a problem hosted on a server (@source), and the server's course macros and version.
• The renderer dictionary passed to individual_webwork_conversion() (and to an ext_converter) now has a macro_dirs entry: the folders of macro files that problems may load. The local renderer's --extraMacroDir arguments are built from the same list, so the two can't drift apart. The server version was already in renderer.

On the CLI side (PreTeXtBook/pretext-cli#1263), each cache entry now also records a stamp. The stamp is a hash of the server's version and of every file in macro_dirs, and an entry whose stamp doesn't match is rebuilt. I repeated your grid = both → grid = none test on the sample chapter through the CLI:

run	time	result
nothing cached	46 s	all 76 rendered
nothing changed	10 s	all 76 from cache
preamble changed	43 s	all 76 rebuilt
nothing changed	10 s	all 76 from cache

A problem hosted on the server, or the course's macros there, can still change without anything on our side noticing. The CLI documents that pretext generate webwork -f is needed after such a change.

2. Behavior changes. No changes.

3. Comment alignment. Fixed at both places.

4. Duplication. Removed the origin loop in webwork_to_xml(), since the check in individual_webwork_conversion() remains. The problem template now receives the assembly identifier as a parameter instead of computing it again.

5. Commit subject. Now WeBWorK:, and short. The three typos are fixed too.

Retesting. I built examples/webwork/sample-chapter against the public server with the earlier version of this PR and with the revision. All 76 representation files, the .pg files, and the 38 images were byte-identical. static-processing="local" remains untested here as well.

Co-authored with Claude Opus 5.5