Enable caching of individual webwork representation files

CHANGELOG.md

@@ -9,6 +9,14 @@ Instructions: Add a subsection under `[Unreleased]` for additions, fixes, change
 
 ## [Unreleased]
 
+### Added
+
+- WeBWorK problems are now cached individually, like `latex-image`, `asymptote`, `sageplot`, and `prefigure` assets.  Each problem's representations file and images are kept in `.cache/webwork/<hash>/`, so when some problems change (or another target with the same source is built), only the changed problems are sent to the WeBWorK server.  A change to the project's PG macro files (including `docinfo/latex-image-preamble`) or to the server's version regenerates every problem.  A problem hosted on the server, or the course's macros there, can change without the CLI noticing; run `pretext generate webwork -f` after such a change.  `pretext prune-cache` removes stale problems too.
+
+### Fixed
+
+- `pretext generate webwork` (or `myopenmath`) on a project without those assets generated no longer generates them twice.
+
 ## [2.56.2] - 2026-11-07
 
 Includes updates to core through commit: [d6a846c](https://github.com/PreTeXtBook/pretext/commit/d6a846cc46399b6e2484eda40c2930bb3c80c5ce)

docs/asset-generation.md

@@ -10,13 +10,17 @@ For each _target_ in the project, we hash the contents of all assets of each typ
 
 The function in core processes the source using an `xsl` template to extract the xml for each asset (this is generally pretty fast).  Then for each extracted asset, it will call the appropriate (often external) routines to convert that source into the desired output format (this can be slow).
 
-However, for select assets (currently `asymptote`, `latex-image`, `prefigure`, and `sageplot`), the CLI intercepts this conversion of individual assets and checks to see if a version of the output is already available in the appropriate subdirectory of `.cache\`.  Specifically, we hash the source of the extracted xml and store `[hash].[ext]` when it is first generated, and if this file exists, we copy the file instead of regenerating the individual asset.
+However, for select assets (currently `asymptote`, `latex-image`, `prefigure`, `sageplot`, and `webwork`), the CLI intercepts this conversion of individual assets and checks to see if a version of the output is already available in the appropriate subdirectory of `.cache\`.  Specifically, we hash the source of the extracted xml and store `[hash].[ext]` when it is first generated, and if this file exists, we copy the file instead of regenerating the individual asset.
+
+A `webwork` problem is a little different.  Its extracted source is the problem's PG code together with the server settings it is rendered under (so changing servers regenerates every problem), and its output is a representations file plus any images the server produced for it.  So its cache entry is a directory, `.cache/webwork/[hash]/`, holding `[id].xml` and `images/[id]-image-*`, and the whole directory is copied back.  Core still contacts the server once per generation to learn its version, but each cached problem is served without a request.  A problem the server could not render gets a placeholder representations file so the build can proceed; the CLI does not cache that placeholder, so the next generation asks the server again.
+
+The extracted source names some inputs without holding them, so its hash alone cannot tell when they change.  Each entry therefore also holds a stamp, `.renderer-stamp`: a hash of the server's version and of the macro files problems may load (the document's own, built from `docinfo` and so carrying `latex-image-preamble`, and any in `external/macros/`).  Core lists those folders for the CLI.  An entry whose stamp differs from the current one is rebuilt; the entry keeps its name, so pruning is unaffected.  Two inputs remain out of the CLI's sight: a problem hosted on the server (`webwork/@source`) and the course's own macro files on the server.  After changing either of those, run `pretext generate webwork -f`.
 
 ### Examples
 
 1. Target `web` is built.  Author edits content but not any asset code.  Target is rebuild, `.cache/.web_assets.json` has current hash for all asset types so no calls to core to generate any assets are made.  The contents of `generated-assets` is unchanged.
 
-2. Target `web` is build.  Author builds target `runestone`, another `format="html"` target with just a slightly different publication file.  No `.cache/.runestone_assets.json` exists, so generation of each asset type is requested from core.  Each `latex-image` asset is just copied from `.cache/latex-image` instead of regenerated.  Assets that we don't have individual caching for (`webwork`, `mermaid`, etc) are regenerated.
+2. Target `web` is build.  Author builds target `runestone`, another `format="html"` target with just a slightly different publication file.  No `.cache/.runestone_assets.json` exists, so generation of each asset type is requested from core.  Each `latex-image` asset is just copied from `.cache/latex-image` instead of regenerated.  Assets that we don't have individual caching for (`mermaid`, `youtube`, etc) are regenerated.
 
 3. Target `web` is built.  Author edits the source of one of five `latex-image` elements.  Author builds `web` again.  The hash in `.cache/.web_assets.json` for key `latex-image` doesn't match, so we request generation of latex-images from core.  For each of the five latex-images, we check whether its source has a hash that matches the stem of an svg in `.cache/latex-image`.  Four of these do, so we copy them.  The fifth doesn't exists in the cache folder, so it is generated by core (and a copy with hash as filename stem is placed into the cache folder).
 

pretext/cli.py

@@ -980,9 +980,10 @@ def generate(
 def prune_cache(ctx: click.Context, dry_run: bool) -> None:
     """
     Remove old versions of generated assets (latex-image, asymptote, sageplot,
-    prefigure) from the cache.  Cached assets are named by a hash of their
-    source, so each edit of an image leaves an old version behind.  This keeps
-    only the cached files matching the current source of some target.
+    prefigure, webwork) from the cache.  Cached assets are named by a hash of
+    their source, so each edit of an image or problem leaves an old version
+    behind.  This keeps only the cached assets matching the current source of
+    some target.
     """
     if utils.cannot_find_project(task="prune the asset cache for"):
         return
@@ -1000,9 +1001,9 @@ def prune_cache(ctx: click.Context, dry_run: bool) -> None:
     for path in stale:
         log.info(f"  {path.relative_to(cache_dir)}")
     if dry_run:
-        log.info(f"{len(stale)} cached file(s) would be removed from {cache_dir}.")
+        log.info(f"{len(stale)} cached asset(s) would be removed from {cache_dir}.")
     else:
-        log.info(f"Removed {len(stale)} cached file(s) from {cache_dir}.")
+        log.info(f"Removed {len(stale)} cached asset(s) from {cache_dir}.")
 
 
 # pretext view

pretext/project/__init__.py

@@ -1353,12 +1353,17 @@ def generate_assets(
             finally:
                 return
 
+        # The command line hands over a tuple; the comparisons below are
+        # with lists.
+        if requested_asset_types is not None:
+            requested_asset_types = list(requested_asset_types)
+
         # clear out the generated assets and cache if requested
         if clean:
             self.clean_assets()
 
         # Ensure that the generated_cache directories exist:
-        for subdir in ["latex-image", "asymptote", "sageplot", "prefigure"]:
+        for subdir in ["latex-image", "asymptote", "sageplot", "prefigure", "webwork"]:
             if not (self.generated_cache_abspath() / subdir).exists():
                 (self.generated_cache_abspath() / subdir).mkdir(
                     parents=True, exist_ok=True
@@ -1457,11 +1462,16 @@ def generate_assets(
                     abort_early=False,
                     dest_dir=(self.generated_dir_abspath() / "webwork").as_posix(),
                     server_params=None,
+                    ext_converter=partial(
+                        generate.individual_webwork,
+                        cache_dir=self.generated_cache_abspath(),
+                        skip_cache=skip_cache,
+                    ),
                 )
                 successful_assets.append("webwork")
             except Exception as e:
                 log.error(
-                    "Unable to generate webwork.  If you already have a webwork-representations.xml file, this might result in unpredictable behavior."
+                    "Unable to generate webwork.  If you already have WeBWorK representation files, this might result in unpredictable behavior."
                 )
                 log.warning(e)
                 log.debug(e, exc_info=True)

pretext/project/generate.py

@@ -5,6 +5,7 @@
 import re
 import shutil
 import tempfile
+from lxml import etree as ET  # noqa: N812
 from .. import core, utils
 
 log = logging.getLogger("ptxlogger")
@@ -219,6 +220,105 @@ def copy_to_cache(output_file: Path, cache_file: Path) -> None:
         shutil.copy2(output_file, tmp_path)
 
 
+def individual_webwork(
+    problem_file: str,
+    dest_dir: str,
+    abort_early: bool,
+    renderer: t.Dict[str, t.Any],
+    cache_dir: Path,
+    skip_cache: bool = False,
+) -> None:
+    """
+    Checks whether cached representations of the WeBWorK problem exist.  If they do, copies them (the representations file and any images) into the dest_dir and returns.  If they do not, calls the core.individual_webwork_conversion function to build them and then copies the files it wrote to the cache_dir.
+    - problem_file is the file core extracts for one problem, named <id>.xml; its hash names the cache entry.
+    - A cache entry is a directory, since a problem's output is a representations file plus any number of images.  It mirrors the layout of dest_dir: <id>.xml at the top, images in images/.
+    - Representations that record a failure to render (static/@failure) are not cached, so that the next generation asks the server again.
+    - The problem file does not hold everything the representations depend on, so each entry also records a stamp (see webwork_renderer_stamp); an entry with a different stamp is rebuilt.
+    """
+    log.debug("Using the CLI's individual_webwork function")
+    asset_file = Path(problem_file).resolve()
+    problem_id = asset_file.stem
+    output_dir = Path(dest_dir).resolve()
+    cache_entry = cache_dir / "webwork" / hash_asset_file(asset_file)
+    cached_reps_file = cache_entry / f"{problem_id}.xml"
+    stamp_file = cache_entry / WEBWORK_STAMP_FILENAME
+    stamp = webwork_renderer_stamp(renderer)
+    if cached_reps_file.exists() and not skip_cache:
+        if stamp_file.exists() and stamp_file.read_text() == stamp:
+            log.debug(
+                f"Copying cached WeBWorK representations {cache_entry} to {output_dir}"
+            )
+            # Files beginning with "." (the stamp, a leftover temporary file) stay in the cache.
+            shutil.copytree(
+                cache_entry,
+                output_dir,
+                dirs_exist_ok=True,
+                ignore=shutil.ignore_patterns(".*"),
+            )
+            return
+        log.debug(
+            f"Cached WeBWorK representations {cache_entry} were built with different macro files or a different server version; rebuilding them."
+        )
+    written_files: t.List[str] = core.individual_webwork_conversion(
+        problem_file, dest_dir, abort_early, renderer
+    )
+    reps_file = output_dir / f"{problem_id}.xml"
+    if not reps_file.exists():
+        msg = f"Failed to generate representations of WeBWorK problem {problem_id}."
+        log.error(msg)
+        raise Exception(msg)
+    if webwork_representations_failed(reps_file):
+        log.debug(
+            f"Not caching the representations of WeBWorK problem {problem_id}, which record a failure to render it."
+        )
+        return
+    log.debug(
+        f"Created WeBWorK representations {reps_file}; saving a copy of the problem's files to cache in {cache_entry}"
+    )
+    if cache_entry.exists():
+        shutil.rmtree(cache_entry)
+    # The representations file is what marks an entry as cached, so it is copied last: an interrupted copy leaves an entry without it, which the next generation replaces.
+    images = [
+        Path(f).resolve() for f in written_files if Path(f).resolve() != reps_file
+    ]
+    for source in images:
+        copy_to_cache(source, cache_entry / source.relative_to(output_dir))
+    with utils.atomic_write(stamp_file) as tmp_path:
+        tmp_path.write_text(stamp)
+    copy_to_cache(reps_file, cached_reps_file)
+    log.debug("Finished individual_webwork function")
+
+
+# The file in each WeBWorK cache entry that records the entry's stamp.  It begins with "." so that it does not match the cache's hash-named files, and so that it is not copied out with the entry.
+WEBWORK_STAMP_FILENAME = ".renderer-stamp"
+
+
+def webwork_renderer_stamp(renderer: t.Dict[str, t.Any]) -> str:
+    """
+    A digest of what a WeBWorK problem's representations depend on beyond the problem file core extracts for it: the version of the server, and the macro files problems may load (the document's own, built from docinfo, and an author's, in the external directory's macros/).  Core lists the macro folders in renderer["macro_dirs"].
+    Not covered, since the CLI cannot see them: a problem hosted on the server (@source), and the course's macro files there.  After changing one of those, generate with --force.
+    """
+    hash = hashlib.md5()
+    for key in ["webwork2_major_version", "webwork2_minor_version"]:
+        hash.update(f"{key}={renderer.get(key)}\n".encode())
+    for macro_dir in renderer.get("macro_dirs", []):
+        for macro_file in sorted(Path(macro_dir).rglob("*")):
+            if macro_file.is_file():
+                hash.update(macro_file.relative_to(macro_dir).as_posix().encode())
+                hash.update(b"\n")
+                hash.update(macro_file.read_bytes())
+                hash.update(b"\n")
+    return hash.hexdigest()
+
+
+def webwork_representations_failed(reps_file: Path) -> bool:
+    """
+    Whether a WeBWorK representations file records that its problem failed to render.  Core writes such a file, marked with static/@failure, in place of the representations so that a build can proceed.
+    """
+    static = ET.parse(reps_file).getroot().find("static")
+    return static is not None and static.get("failure") is not None
+
+
 def hash_asset_file(asset_file: Path) -> str:
     hash = hashlib.md5()
     # hash the asset file
@@ -231,7 +331,7 @@ def cache_asset_filename(asset_file: Path, extension: str, cache_dir: Path) -> P
     return cache_dir / f"{hash_asset_file(asset_file)}.{extension}"
 
 
-# Cached asset types, with the core extraction stylesheet that produces the source files whose hashes name the cached files.  Each entry is a list of extra stringparams to run the extraction with; sageplot extraction differs by output format, so it runs once per format.
+# Cached asset types, with the core extraction stylesheet that produces the source files whose hashes name the cache entries.  Each entry is a list of extra stringparams to run the extraction with; sageplot extraction differs by output format, so it runs once per format.
 CACHED_ASSET_EXTRACTIONS: t.Dict[str, t.Tuple[str, t.List[t.Dict[str, str]]]] = {
     "latex-image": ("extract-latex-image.xsl", [{}]),
     "asymptote": ("extract-asymptote.xsl", [{}]),
@@ -240,11 +340,14 @@ def cache_asset_filename(asset_file: Path, extension: str, cache_dir: Path) -> P
         [{"sageplot.fileformat": f} for f in ["pdf", "svg", "png", "html"]],
     ),
     "prefigure": ("extract-prefigure.xsl", [{}]),
+    "webwork": ("extract-pg.xsl", [{}]),
 }
 
 
 # Names of cache files, as created by cache_asset_filename (plus prefigure's -annotations.xml companions): an md5 hex digest, an optional -annotations suffix, and an extension.
 CACHE_FILENAME_PATTERN = re.compile(r"^([0-9a-f]{32})(?:-annotations)?\.[A-Za-z0-9]+$")
+# Names of cache directories, as created by individual_webwork: a bare md5 hex digest, holding every file of one asset.
+CACHE_DIRNAME_PATTERN = re.compile(r"^[0-9a-f]{32}$")
 
 
 def extracted_asset_hashes(
@@ -270,29 +373,54 @@ def extracted_asset_hashes(
     return hashes
 
 
-def is_stale(cache_file: Path, live_hashes: t.Set[str]) -> bool:
+def cache_entry_hash(cache_entry: Path) -> t.Optional[str]:
+    """
+    The hash naming a cache entry: a file such as <hash>.svg or <hash>-annotations.xml, or a directory <hash> holding every file of one asset.  None for anything else.
+    """
+    if cache_entry.is_dir():
+        match = CACHE_DIRNAME_PATTERN.match(cache_entry.name)
+        return cache_entry.name if match else None
+    match = CACHE_FILENAME_PATTERN.match(cache_entry.name)
+    return match.group(1) if match else None
+
+
+def is_stale(cache_entry: Path, live_hashes: t.Set[str]) -> bool:
     """
-    Decides whether a file in one of the cached asset directories should be removed, given the hashes of the asset sources currently in the project.  Only files named by a hash (e.g. <hash>.svg or <hash>-annotations.xml) are candidates; anything else is left alone.  A live hash keeps all of its formats.
+    Decides whether an entry in one of the cached asset directories should be removed, given the hashes of the asset sources currently in the project.  Only entries named by a hash are candidates; anything else is left alone.  A live hash keeps all of its formats.
     """
-    match = CACHE_FILENAME_PATTERN.match(cache_file.name)
-    return match is not None and match.group(1) not in live_hashes
+    hash = cache_entry_hash(cache_entry)
+    return hash is not None and hash not in live_hashes
+
+
+def stale_cache_entries(directory: Path, live_hashes: t.Set[str]) -> t.List[Path]:
+    """
+    The stale entries under a cached asset directory, descending into subdirectories such as prefigure/tactile but not into a stale entry, which is removed whole.
+    """
+    stale: t.List[Path] = []
+    for cache_entry in sorted(directory.iterdir()):
+        if is_stale(cache_entry, live_hashes):
+            stale.append(cache_entry)
+        elif cache_entry.is_dir():
+            stale.extend(stale_cache_entries(cache_entry, live_hashes))
+    return stale
 
 
 def prune_cache(
     cache_dir: Path, live_hashes: t.Dict[str, t.Set[str]], dry_run: bool = False
 ) -> t.List[Path]:
     """
-    Removes cached asset files whose source no longer appears in the project.  Returns the list of removed (or, with dry_run, removable) files.
+    Removes cached assets whose source no longer appears in the project.  Returns the list of removed (or, with dry_run, removable) files and directories.
     """
     stale: t.List[Path] = []
     for asset_type, hashes in live_hashes.items():
         asset_cache_dir = cache_dir / asset_type
         if not asset_cache_dir.is_dir():
             continue
-        # rglob to include subdirectories such as prefigure/tactile
-        for cache_file in sorted(asset_cache_dir.rglob("*")):
-            if cache_file.is_file() and is_stale(cache_file, hashes):
-                stale.append(cache_file)
-                if not dry_run:
-                    cache_file.unlink()
+        for cache_entry in stale_cache_entries(asset_cache_dir, hashes):
+            stale.append(cache_entry)
+            if not dry_run:
+                if cache_entry.is_dir():
+                    shutil.rmtree(cache_entry)
+                else:
+                    cache_entry.unlink()
     return stale