diff --git a/CHANGELOG.md b/CHANGELOG.md index 101087ad..f9889de6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,14 @@ Instructions: Add a subsection under `[Unreleased]` for additions, fixes, change ## [Unreleased] +### Added + +- WeBWorK problems are now cached individually, like `latex-image`, `asymptote`, `sageplot`, and `prefigure` assets. Each problem's representations file and images are kept in `.cache/webwork//`, so when some problems change (or another target with the same source is built), only the changed problems are sent to the WeBWorK server. A change to the project's PG macro files (including `docinfo/latex-image-preamble`) or to the server's version regenerates every problem. A problem hosted on the server, or the course's macros there, can change without the CLI noticing; run `pretext generate webwork -f` after such a change. `pretext prune-cache` removes stale problems too. + +### Fixed + +- `pretext generate webwork` (or `myopenmath`) on a project without those assets generated no longer generates them twice. + ## [2.56.2] - 2026-11-07 Includes updates to core through commit: [d6a846c](https://github.com/PreTeXtBook/pretext/commit/d6a846cc46399b6e2484eda40c2930bb3c80c5ce) diff --git a/docs/asset-generation.md b/docs/asset-generation.md index f47d50f6..ac5d6f56 100644 --- a/docs/asset-generation.md +++ b/docs/asset-generation.md @@ -10,13 +10,17 @@ For each _target_ in the project, we hash the contents of all assets of each typ The function in core processes the source using an `xsl` template to extract the xml for each asset (this is generally pretty fast). Then for each extracted asset, it will call the appropriate (often external) routines to convert that source into the desired output format (this can be slow). -However, for select assets (currently `asymptote`, `latex-image`, `prefigure`, and `sageplot`), the CLI intercepts this conversion of individual assets and checks to see if a version of the output is already available in the appropriate subdirectory of `.cache\`. Specifically, we hash the source of the extracted xml and store `[hash].[ext]` when it is first generated, and if this file exists, we copy the file instead of regenerating the individual asset. +However, for select assets (currently `asymptote`, `latex-image`, `prefigure`, `sageplot`, and `webwork`), the CLI intercepts this conversion of individual assets and checks to see if a version of the output is already available in the appropriate subdirectory of `.cache\`. Specifically, we hash the source of the extracted xml and store `[hash].[ext]` when it is first generated, and if this file exists, we copy the file instead of regenerating the individual asset. + +A `webwork` problem is a little different. Its extracted source is the problem's PG code together with the server settings it is rendered under (so changing servers regenerates every problem), and its output is a representations file plus any images the server produced for it. So its cache entry is a directory, `.cache/webwork/[hash]/`, holding `[id].xml` and `images/[id]-image-*`, and the whole directory is copied back. Core still contacts the server once per generation to learn its version, but each cached problem is served without a request. A problem the server could not render gets a placeholder representations file so the build can proceed; the CLI does not cache that placeholder, so the next generation asks the server again. + +The extracted source names some inputs without holding them, so its hash alone cannot tell when they change. Each entry therefore also holds a stamp, `.renderer-stamp`: a hash of the server's version and of the macro files problems may load (the document's own, built from `docinfo` and so carrying `latex-image-preamble`, and any in `external/macros/`). Core lists those folders for the CLI. An entry whose stamp differs from the current one is rebuilt; the entry keeps its name, so pruning is unaffected. Two inputs remain out of the CLI's sight: a problem hosted on the server (`webwork/@source`) and the course's own macro files on the server. After changing either of those, run `pretext generate webwork -f`. ### Examples 1. Target `web` is built. Author edits content but not any asset code. Target is rebuild, `.cache/.web_assets.json` has current hash for all asset types so no calls to core to generate any assets are made. The contents of `generated-assets` is unchanged. -2. Target `web` is build. Author builds target `runestone`, another `format="html"` target with just a slightly different publication file. No `.cache/.runestone_assets.json` exists, so generation of each asset type is requested from core. Each `latex-image` asset is just copied from `.cache/latex-image` instead of regenerated. Assets that we don't have individual caching for (`webwork`, `mermaid`, etc) are regenerated. +2. Target `web` is build. Author builds target `runestone`, another `format="html"` target with just a slightly different publication file. No `.cache/.runestone_assets.json` exists, so generation of each asset type is requested from core. Each `latex-image` asset is just copied from `.cache/latex-image` instead of regenerated. Assets that we don't have individual caching for (`mermaid`, `youtube`, etc) are regenerated. 3. Target `web` is built. Author edits the source of one of five `latex-image` elements. Author builds `web` again. The hash in `.cache/.web_assets.json` for key `latex-image` doesn't match, so we request generation of latex-images from core. For each of the five latex-images, we check whether its source has a hash that matches the stem of an svg in `.cache/latex-image`. Four of these do, so we copy them. The fifth doesn't exists in the cache folder, so it is generated by core (and a copy with hash as filename stem is placed into the cache folder). diff --git a/pretext/cli.py b/pretext/cli.py index 73b3fb59..bf7f1102 100644 --- a/pretext/cli.py +++ b/pretext/cli.py @@ -980,9 +980,10 @@ def generate( def prune_cache(ctx: click.Context, dry_run: bool) -> None: """ Remove old versions of generated assets (latex-image, asymptote, sageplot, - prefigure) from the cache. Cached assets are named by a hash of their - source, so each edit of an image leaves an old version behind. This keeps - only the cached files matching the current source of some target. + prefigure, webwork) from the cache. Cached assets are named by a hash of + their source, so each edit of an image or problem leaves an old version + behind. This keeps only the cached assets matching the current source of + some target. """ if utils.cannot_find_project(task="prune the asset cache for"): return @@ -1000,9 +1001,9 @@ def prune_cache(ctx: click.Context, dry_run: bool) -> None: for path in stale: log.info(f" {path.relative_to(cache_dir)}") if dry_run: - log.info(f"{len(stale)} cached file(s) would be removed from {cache_dir}.") + log.info(f"{len(stale)} cached asset(s) would be removed from {cache_dir}.") else: - log.info(f"Removed {len(stale)} cached file(s) from {cache_dir}.") + log.info(f"Removed {len(stale)} cached asset(s) from {cache_dir}.") # pretext view diff --git a/pretext/project/__init__.py b/pretext/project/__init__.py index a27d4834..aabee6b3 100644 --- a/pretext/project/__init__.py +++ b/pretext/project/__init__.py @@ -1353,12 +1353,17 @@ def generate_assets( finally: return + # The command line hands over a tuple; the comparisons below are + # with lists. + if requested_asset_types is not None: + requested_asset_types = list(requested_asset_types) + # clear out the generated assets and cache if requested if clean: self.clean_assets() # Ensure that the generated_cache directories exist: - for subdir in ["latex-image", "asymptote", "sageplot", "prefigure"]: + for subdir in ["latex-image", "asymptote", "sageplot", "prefigure", "webwork"]: if not (self.generated_cache_abspath() / subdir).exists(): (self.generated_cache_abspath() / subdir).mkdir( parents=True, exist_ok=True @@ -1457,11 +1462,16 @@ def generate_assets( abort_early=False, dest_dir=(self.generated_dir_abspath() / "webwork").as_posix(), server_params=None, + ext_converter=partial( + generate.individual_webwork, + cache_dir=self.generated_cache_abspath(), + skip_cache=skip_cache, + ), ) successful_assets.append("webwork") except Exception as e: log.error( - "Unable to generate webwork. If you already have a webwork-representations.xml file, this might result in unpredictable behavior." + "Unable to generate webwork. If you already have WeBWorK representation files, this might result in unpredictable behavior." ) log.warning(e) log.debug(e, exc_info=True) diff --git a/pretext/project/generate.py b/pretext/project/generate.py index af99df38..a7853041 100644 --- a/pretext/project/generate.py +++ b/pretext/project/generate.py @@ -5,6 +5,7 @@ import re import shutil import tempfile +from lxml import etree as ET # noqa: N812 from .. import core, utils log = logging.getLogger("ptxlogger") @@ -219,6 +220,105 @@ def copy_to_cache(output_file: Path, cache_file: Path) -> None: shutil.copy2(output_file, tmp_path) +def individual_webwork( + problem_file: str, + dest_dir: str, + abort_early: bool, + renderer: t.Dict[str, t.Any], + cache_dir: Path, + skip_cache: bool = False, +) -> None: + """ + Checks whether cached representations of the WeBWorK problem exist. If they do, copies them (the representations file and any images) into the dest_dir and returns. If they do not, calls the core.individual_webwork_conversion function to build them and then copies the files it wrote to the cache_dir. + - problem_file is the file core extracts for one problem, named .xml; its hash names the cache entry. + - A cache entry is a directory, since a problem's output is a representations file plus any number of images. It mirrors the layout of dest_dir: .xml at the top, images in images/. + - Representations that record a failure to render (static/@failure) are not cached, so that the next generation asks the server again. + - The problem file does not hold everything the representations depend on, so each entry also records a stamp (see webwork_renderer_stamp); an entry with a different stamp is rebuilt. + """ + log.debug("Using the CLI's individual_webwork function") + asset_file = Path(problem_file).resolve() + problem_id = asset_file.stem + output_dir = Path(dest_dir).resolve() + cache_entry = cache_dir / "webwork" / hash_asset_file(asset_file) + cached_reps_file = cache_entry / f"{problem_id}.xml" + stamp_file = cache_entry / WEBWORK_STAMP_FILENAME + stamp = webwork_renderer_stamp(renderer) + if cached_reps_file.exists() and not skip_cache: + if stamp_file.exists() and stamp_file.read_text() == stamp: + log.debug( + f"Copying cached WeBWorK representations {cache_entry} to {output_dir}" + ) + # Files beginning with "." (the stamp, a leftover temporary file) stay in the cache. + shutil.copytree( + cache_entry, + output_dir, + dirs_exist_ok=True, + ignore=shutil.ignore_patterns(".*"), + ) + return + log.debug( + f"Cached WeBWorK representations {cache_entry} were built with different macro files or a different server version; rebuilding them." + ) + written_files: t.List[str] = core.individual_webwork_conversion( + problem_file, dest_dir, abort_early, renderer + ) + reps_file = output_dir / f"{problem_id}.xml" + if not reps_file.exists(): + msg = f"Failed to generate representations of WeBWorK problem {problem_id}." + log.error(msg) + raise Exception(msg) + if webwork_representations_failed(reps_file): + log.debug( + f"Not caching the representations of WeBWorK problem {problem_id}, which record a failure to render it." + ) + return + log.debug( + f"Created WeBWorK representations {reps_file}; saving a copy of the problem's files to cache in {cache_entry}" + ) + if cache_entry.exists(): + shutil.rmtree(cache_entry) + # The representations file is what marks an entry as cached, so it is copied last: an interrupted copy leaves an entry without it, which the next generation replaces. + images = [ + Path(f).resolve() for f in written_files if Path(f).resolve() != reps_file + ] + for source in images: + copy_to_cache(source, cache_entry / source.relative_to(output_dir)) + with utils.atomic_write(stamp_file) as tmp_path: + tmp_path.write_text(stamp) + copy_to_cache(reps_file, cached_reps_file) + log.debug("Finished individual_webwork function") + + +# The file in each WeBWorK cache entry that records the entry's stamp. It begins with "." so that it does not match the cache's hash-named files, and so that it is not copied out with the entry. +WEBWORK_STAMP_FILENAME = ".renderer-stamp" + + +def webwork_renderer_stamp(renderer: t.Dict[str, t.Any]) -> str: + """ + A digest of what a WeBWorK problem's representations depend on beyond the problem file core extracts for it: the version of the server, and the macro files problems may load (the document's own, built from docinfo, and an author's, in the external directory's macros/). Core lists the macro folders in renderer["macro_dirs"]. + Not covered, since the CLI cannot see them: a problem hosted on the server (@source), and the course's macro files there. After changing one of those, generate with --force. + """ + hash = hashlib.md5() + for key in ["webwork2_major_version", "webwork2_minor_version"]: + hash.update(f"{key}={renderer.get(key)}\n".encode()) + for macro_dir in renderer.get("macro_dirs", []): + for macro_file in sorted(Path(macro_dir).rglob("*")): + if macro_file.is_file(): + hash.update(macro_file.relative_to(macro_dir).as_posix().encode()) + hash.update(b"\n") + hash.update(macro_file.read_bytes()) + hash.update(b"\n") + return hash.hexdigest() + + +def webwork_representations_failed(reps_file: Path) -> bool: + """ + Whether a WeBWorK representations file records that its problem failed to render. Core writes such a file, marked with static/@failure, in place of the representations so that a build can proceed. + """ + static = ET.parse(reps_file).getroot().find("static") + return static is not None and static.get("failure") is not None + + def hash_asset_file(asset_file: Path) -> str: hash = hashlib.md5() # hash the asset file @@ -231,7 +331,7 @@ def cache_asset_filename(asset_file: Path, extension: str, cache_dir: Path) -> P return cache_dir / f"{hash_asset_file(asset_file)}.{extension}" -# Cached asset types, with the core extraction stylesheet that produces the source files whose hashes name the cached files. Each entry is a list of extra stringparams to run the extraction with; sageplot extraction differs by output format, so it runs once per format. +# Cached asset types, with the core extraction stylesheet that produces the source files whose hashes name the cache entries. Each entry is a list of extra stringparams to run the extraction with; sageplot extraction differs by output format, so it runs once per format. CACHED_ASSET_EXTRACTIONS: t.Dict[str, t.Tuple[str, t.List[t.Dict[str, str]]]] = { "latex-image": ("extract-latex-image.xsl", [{}]), "asymptote": ("extract-asymptote.xsl", [{}]), @@ -240,11 +340,14 @@ def cache_asset_filename(asset_file: Path, extension: str, cache_dir: Path) -> P [{"sageplot.fileformat": f} for f in ["pdf", "svg", "png", "html"]], ), "prefigure": ("extract-prefigure.xsl", [{}]), + "webwork": ("extract-pg.xsl", [{}]), } # Names of cache files, as created by cache_asset_filename (plus prefigure's -annotations.xml companions): an md5 hex digest, an optional -annotations suffix, and an extension. CACHE_FILENAME_PATTERN = re.compile(r"^([0-9a-f]{32})(?:-annotations)?\.[A-Za-z0-9]+$") +# Names of cache directories, as created by individual_webwork: a bare md5 hex digest, holding every file of one asset. +CACHE_DIRNAME_PATTERN = re.compile(r"^[0-9a-f]{32}$") def extracted_asset_hashes( @@ -270,29 +373,54 @@ def extracted_asset_hashes( return hashes -def is_stale(cache_file: Path, live_hashes: t.Set[str]) -> bool: +def cache_entry_hash(cache_entry: Path) -> t.Optional[str]: + """ + The hash naming a cache entry: a file such as .svg or -annotations.xml, or a directory holding every file of one asset. None for anything else. + """ + if cache_entry.is_dir(): + match = CACHE_DIRNAME_PATTERN.match(cache_entry.name) + return cache_entry.name if match else None + match = CACHE_FILENAME_PATTERN.match(cache_entry.name) + return match.group(1) if match else None + + +def is_stale(cache_entry: Path, live_hashes: t.Set[str]) -> bool: """ - Decides whether a file in one of the cached asset directories should be removed, given the hashes of the asset sources currently in the project. Only files named by a hash (e.g. .svg or -annotations.xml) are candidates; anything else is left alone. A live hash keeps all of its formats. + Decides whether an entry in one of the cached asset directories should be removed, given the hashes of the asset sources currently in the project. Only entries named by a hash are candidates; anything else is left alone. A live hash keeps all of its formats. """ - match = CACHE_FILENAME_PATTERN.match(cache_file.name) - return match is not None and match.group(1) not in live_hashes + hash = cache_entry_hash(cache_entry) + return hash is not None and hash not in live_hashes + + +def stale_cache_entries(directory: Path, live_hashes: t.Set[str]) -> t.List[Path]: + """ + The stale entries under a cached asset directory, descending into subdirectories such as prefigure/tactile but not into a stale entry, which is removed whole. + """ + stale: t.List[Path] = [] + for cache_entry in sorted(directory.iterdir()): + if is_stale(cache_entry, live_hashes): + stale.append(cache_entry) + elif cache_entry.is_dir(): + stale.extend(stale_cache_entries(cache_entry, live_hashes)) + return stale def prune_cache( cache_dir: Path, live_hashes: t.Dict[str, t.Set[str]], dry_run: bool = False ) -> t.List[Path]: """ - Removes cached asset files whose source no longer appears in the project. Returns the list of removed (or, with dry_run, removable) files. + Removes cached assets whose source no longer appears in the project. Returns the list of removed (or, with dry_run, removable) files and directories. """ stale: t.List[Path] = [] for asset_type, hashes in live_hashes.items(): asset_cache_dir = cache_dir / asset_type if not asset_cache_dir.is_dir(): continue - # rglob to include subdirectories such as prefigure/tactile - for cache_file in sorted(asset_cache_dir.rglob("*")): - if cache_file.is_file() and is_stale(cache_file, hashes): - stale.append(cache_file) - if not dry_run: - cache_file.unlink() + for cache_entry in stale_cache_entries(asset_cache_dir, hashes): + stale.append(cache_entry) + if not dry_run: + if cache_entry.is_dir(): + shutil.rmtree(cache_entry) + else: + cache_entry.unlink() return stale diff --git a/tests/test_generate.py b/tests/test_generate.py index 25761a7c..44bcc802 100644 --- a/tests/test_generate.py +++ b/tests/test_generate.py @@ -6,6 +6,8 @@ exception so that the asset type is not added to the generated asset cache. """ +import shutil +import typing as t from pathlib import Path from unittest.mock import patch @@ -388,3 +390,293 @@ def build() -> None: asset_file, "svg", tmp_path / "cache" / "asymptote" ) assert cache_file.read_text() == "complete" + + +# --------------------------------------------------------------------------- +# WeBWorK: a cache entry is a directory holding the representations file and +# the problem's images. + + +def _write_representations(reps_file: Path, failure: t.Optional[str] = None) -> None: + attribute = f' failure="{failure}"' if failure else "" + reps_file.write_text( + f'' + ) + + +def _fake_webwork_conversion( + problem_file: str, dest_dir: str, abort_early: bool, renderer: t.Dict[str, t.Any] +) -> t.List[str]: + """Simulates core: writes the representations file and one image, returning both.""" + problem_id = Path(problem_file).stem + reps_file = Path(dest_dir) / f"{problem_id}.xml" + image_file = Path(dest_dir) / "images" / f"{problem_id}-image-1.png" + image_file.parent.mkdir(exist_ok=True) + _write_representations(reps_file) + image_file.write_bytes(b"png") + return [str(reps_file), str(image_file)] + + +def _webwork_dirs(tmp_path: Path) -> t.Tuple[Path, Path, Path]: + """The extracted problem file, the destination directory, and the cache directory.""" + problem_file = tmp_path / "extracted" / "ex-graph.xml" + problem_file.parent.mkdir() + problem_file.write_text("") + dest_dir = tmp_path / "generated" / "webwork" + (dest_dir / "images").mkdir(parents=True) + cache_dir = tmp_path / "cache" + (cache_dir / "webwork").mkdir(parents=True) + return problem_file, dest_dir, cache_dir + + +def test_individual_webwork_raises_on_missing_output(tmp_path: Path) -> None: + """individual_webwork should raise when core does not write the representations file.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + return_value=[], + ): + with pytest.raises(Exception, match="Failed to generate representations"): + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir, skip_cache=True + ) + + +def test_individual_webwork_caches_every_written_file(tmp_path: Path) -> None: + """The files core reports writing are copied into a hash-named cache directory, mirroring the destination layout.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=_fake_webwork_conversion, + ): + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + cache_entry = cache_dir / "webwork" / generate.hash_asset_file(problem_file) + assert (cache_entry / "ex-graph.xml").exists() + assert (cache_entry / "images" / "ex-graph-image-1.png").exists() + + +def test_individual_webwork_uses_cache_without_calling_core(tmp_path: Path) -> None: + """A second generation of an unchanged problem restores its files from the cache and never calls core.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=_fake_webwork_conversion, + ): + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + shutil.rmtree(dest_dir) + (dest_dir / "images").mkdir(parents=True) + with patch( + "pretext.project.generate.core.individual_webwork_conversion" + ) as conversion: + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + conversion.assert_not_called() + assert (dest_dir / "ex-graph.xml").exists() + assert (dest_dir / "images" / "ex-graph-image-1.png").read_bytes() == b"png" + + +def test_individual_webwork_skip_cache_regenerates(tmp_path: Path) -> None: + """With skip_cache, core is called even though a cache entry exists, and the entry is replaced.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + cache_entry = cache_dir / "webwork" / generate.hash_asset_file(problem_file) + cache_entry.mkdir() + (cache_entry / "ex-graph.xml").write_text("stale") + (cache_entry / "images").mkdir() + (cache_entry / "images" / "ex-graph-image-2.png").write_bytes(b"old") + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=_fake_webwork_conversion, + ) as conversion: + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir, skip_cache=True + ) + conversion.assert_called_once() + assert (cache_entry / "ex-graph.xml").read_text() != "stale" + assert not (cache_entry / "images" / "ex-graph-image-2.png").exists() + assert (cache_entry / "images" / "ex-graph-image-1.png").exists() + + +def test_individual_webwork_does_not_cache_failures(tmp_path: Path) -> None: + """Representations recording a failure to render are written for the build but not cached.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + + def failed_conversion( + problem_file: str, + dest_dir: str, + abort_early: bool, + renderer: t.Dict[str, t.Any], + ) -> t.List[str]: + reps_file = Path(dest_dir) / f"{Path(problem_file).stem}.xml" + _write_representations(reps_file, failure="compile") + return [str(reps_file)] + + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=failed_conversion, + ): + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + assert (dest_dir / "ex-graph.xml").exists() + assert not list((cache_dir / "webwork").iterdir()) + + +def test_is_stale_directories(tmp_path: Path) -> None: + """A hash-named directory is a cache entry; other directories are not.""" + for name in [LIVE, STALE, "images", "tactile"]: + (tmp_path / name).mkdir() + assert generate.is_stale(tmp_path / LIVE, {LIVE}) is False + assert generate.is_stale(tmp_path / STALE, {LIVE}) is True + assert generate.is_stale(tmp_path / "images", {LIVE}) is False + assert generate.is_stale(tmp_path / "tactile", {LIVE}) is False + + +def test_prune_cache_removes_stale_webwork_directories(tmp_path: Path) -> None: + """A stale webwork entry is removed whole, as one entry; a live one keeps all its files.""" + for name in [ + f"webwork/{LIVE}/ex-live.xml", + f"webwork/{LIVE}/images/ex-live-image-1.png", + f"webwork/{STALE}/ex-stale.xml", + f"webwork/{STALE}/images/ex-stale-image-1.png", + ]: + path = tmp_path / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("x") + removed = generate.prune_cache(tmp_path, {"webwork": {LIVE}}) + assert removed == [tmp_path / "webwork" / STALE] + assert not (tmp_path / "webwork" / STALE).exists() + assert (tmp_path / "webwork" / LIVE / "images" / "ex-live-image-1.png").exists() + + +def test_interrupted_webwork_cache_copy_regenerates_on_next_build( + tmp_path: Path, +) -> None: + """If saving a problem's files to the cache is interrupted, the entry lacks its representations file, so the next generation calls core again instead of restoring a partial entry.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=_fake_webwork_conversion, + ) as conversion: + # interrupt the second copy: the image is cached, the representations file is not + real_copy2 = shutil.copy2 + copies: t.List[Path] = [] + + def copy_then_interrupt(src: Path, dst: Path) -> None: + copies.append(Path(src)) + if len(copies) == 2: + _interrupted_copy(src, dst) + real_copy2(src, dst) + + with patch( + "pretext.project.generate.shutil.copy2", side_effect=copy_then_interrupt + ): + with pytest.raises(KeyboardInterrupt): + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + assert copies[1].name == "ex-graph.xml" + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + assert conversion.call_count == 2 + cache_entry = cache_dir / "webwork" / generate.hash_asset_file(problem_file) + assert (cache_entry / "ex-graph.xml").exists() + assert (cache_entry / "images" / "ex-graph-image-1.png").exists() + + +def _generate_webwork_twice( + tmp_path: Path, + first_renderer: t.Dict[str, t.Any], + second_renderer: t.Dict[str, t.Any], + between: t.Callable[[], t.Any] = lambda: None, +) -> int: + """Generates one problem twice, with the given renderers; returns how many times core was called.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=_fake_webwork_conversion, + ) as conversion: + generate.individual_webwork( + str(problem_file), str(dest_dir), False, first_renderer, cache_dir + ) + between() + generate.individual_webwork( + str(problem_file), str(dest_dir), False, second_renderer, cache_dir + ) + return conversion.call_count + + +def test_individual_webwork_rebuilds_when_a_macro_file_changes( + tmp_path: Path, +) -> None: + """The problem file names macro files without holding them, so a change to one invalidates the cache entry.""" + macro_dir = tmp_path / "macros" + macro_dir.mkdir() + macro_file = macro_dir / "BOOK.pl" + macro_file.write_text("sub latexImagePreamble { 'grid = both' }") + renderer = {"macro_dirs": [str(macro_dir)]} + calls = _generate_webwork_twice( + tmp_path, + renderer, + renderer, + between=lambda: macro_file.write_text( + "sub latexImagePreamble { 'grid = none' }" + ), + ) + assert calls == 2 + + +def test_individual_webwork_uses_cache_with_unchanged_macro_files( + tmp_path: Path, +) -> None: + """Macro files that have not changed leave the cache entry usable.""" + macro_dir = tmp_path / "macros" + macro_dir.mkdir() + (macro_dir / "BOOK.pl").write_text("1;") + renderer = {"macro_dirs": [str(macro_dir)]} + assert _generate_webwork_twice(tmp_path, renderer, renderer) == 1 + + +def test_individual_webwork_rebuilds_when_the_server_version_changes( + tmp_path: Path, +) -> None: + """The server's version is written into every representations file, so a new version invalidates the cache entry.""" + calls = _generate_webwork_twice( + tmp_path, + {"webwork2_major_version": 2, "webwork2_minor_version": 19}, + {"webwork2_major_version": 2, "webwork2_minor_version": 20}, + ) + assert calls == 2 + + +def test_individual_webwork_stamp_stays_in_cache(tmp_path: Path) -> None: + """The stamp recorded in a cache entry is not copied into the destination with the problem's files.""" + problem_file, dest_dir, cache_dir = _webwork_dirs(tmp_path) + with patch( + "pretext.project.generate.core.individual_webwork_conversion", + side_effect=_fake_webwork_conversion, + ): + generate.individual_webwork( + str(problem_file), str(dest_dir), False, {}, cache_dir + ) + shutil.rmtree(dest_dir) + (dest_dir / "images").mkdir(parents=True) + generate.individual_webwork(str(problem_file), str(dest_dir), False, {}, cache_dir) + cache_entry = cache_dir / "webwork" / generate.hash_asset_file(problem_file) + assert (cache_entry / generate.WEBWORK_STAMP_FILENAME).exists() + assert not (dest_dir / generate.WEBWORK_STAMP_FILENAME).exists() + assert (dest_dir / "ex-graph.xml").exists() + + +def test_prune_cache_keeps_the_stamp_of_a_live_webwork_entry(tmp_path: Path) -> None: + """The stamp is not named by a hash, so pruning leaves it in a live entry.""" + stamp = tmp_path / "webwork" / LIVE / generate.WEBWORK_STAMP_FILENAME + stamp.parent.mkdir(parents=True) + stamp.write_text("x") + assert generate.prune_cache(tmp_path, {"webwork": {LIVE}}) == [] + assert stamp.exists()