Script: upgrade MathJax to version 4

[rbeezer]

Upgrades script/mjsre/mj-sre-page.js from mathjax-full 3.2.2 + Speech Rule Engine 4.0.7 to @mathjax/src 4.1.3 (with the mathjax-newcm font) + Speech Rule Engine 5.0.0-rc.4. The command-line interface is unchanged, so the mathjax_latex() pipeline needs no adjustment.

Version 4 differences handled in the rewrite:

• No AllPackages module: the package list comes from MathJax's component map, and every listed extension is loaded, excluding autoload/require/colorv2 (as version 3 did) and the new bbm/bboldx/dsfont/physics, which redefine core macros (version 4's physics turns \div into a divergence).
• The TeX input stamps data-latex attributes on every node; they remain in the internal MathML for the Speech Rule Engine's future use, but are filtered from every output (they bloat outputs, and a raw < inside one breaks the XML post-processing).
• In-line line-breaking is on by default and splits math into several SVG fragments; disabled.
• An author's explicit \\ inside in-line math becomes a rendered line break with a percentage-width SVG; the SVG rendering ignores it, as version 3 did, while MathML, speech, and braille retain it.

Testing. All four representation formats (mml, svg, speech, nemeth) were generated for the sample article (670 math items), the sample book (2,972), and AATA (16,216) under both versions, and every difference classified. Changes come in identifiable clusters: SRE 5 speech refinements ("normal infinity" → "infinity", "c i s" → "cis"), revised Nemeth rule sets (6–8% of items re-encoded), MathML annotation changes with content differences limited to added invisible function-application characters and codepoint normalizations, and SVG dimensions agreeing within 3.6ex except framed arrays (~2.3ex wider). Side-by-side browser renderings of extracted SVG pairs are visually indistinguishable. Full pdf-fo (308 pages) and epub-svg builds of the sample article complete cleanly on the new version.

Note: Speech Rule Engine is pinned to 5.0.0-rc.4, matching MathJax's own dependency; the pin should move to 5.0 final when released.

Claude Fable 5, acting as a coding assistant for Rob Beezer

[rbeezer]

Making this available for comment prior to merging the changes. Replicating v3 behavior now, and we can evaluate things like line-breaking and the physics package once this is integrated and in use for a while.

Line-breaking may be very useful for the new "LaTeX No LaTeX" project making accessible PDFs via XSL-FO ("Formatting Objects"). Since long displays currently become a single SVG image and these are not currently page-breaking.

Pinging: @zorkow, @dpvc, @davidaustinm

[dpvc]

I've taken a look at this code, though (full disclosure) I haven't actually tried running it. I suspect it does work as is, but I give some suggestions for changes that use more of the v4 features and list some caveats to the approach you are taking.

[dpvc]

The documentation suggests another approach to this:

import {source} from '@mathjax/src/components/js/source.js';
const AllPackages = Object.keys(source).filter((name) => name.substring(0,6) === '[tex]/').sort();

You can filter out the other packages you want to remove as well. Also, using

const excluded = new Set(['autoload', 'require', 'colorv2', 'bbm', 'bboldx', 'dsfont', 'physics']);

with excluded.has(name) might be a bit more efficient, as the set will be set up to handle queries efficiently.

It's also possible to set up your app to handle dynamically loaded components, like those that are autoloaded or that use \require{} explicitly. That way you don't have to load everything up front, when it might not be needed. There is an example in the v4 documentation. There are also updated examples in the MathJax node demos repository, in the 'mixed' subdirectory of cjs (for CommonJS usage, as is your case) or mjs (for ES6 apps).

[dpvc]

You might consider using the ui/Menu/MmlVisitor.js instead, as it has options for not including the data-latex and data-semantic attributes. You are currently removing the LaTeX attributes before doing the enrichment and before generating speech. Although SRE doesn't currently use the LaTeX attributes during the enrichment and speech-generation processes, we have plans to improve the results by taking those attributes into account. Removing them early will prevent that future feature from working. Using the MmlVisitor will allow you to control the attributes that are presented in the MathML it generates, while still allowing SRE to have full access to the attributes.

As for the SVG output, one could do

const {CommonWrapper} = require('@mathjax/src/js/output/common/Wrapper.js');
CommonWrapper.skipAttributes['data-latex']= true;

would prevent the data-latex attribute from being added to the SVG or CHTML output nodes. To remove the data-semantic attributes, you would need to add all the ones you want to remove.

[dpvc]

See above comment about using the menu's MmlVisitor rather than modifying the internal data itself.

Also, MathJax already has a tree walker, so you don't have to write one yourself. For example,

function removeLatexAttributes(node) {
  node.walkTree((node) => node.attributes.unset('data-latex');
}

would do it. The tree walker doesn't get called on the text nodes, so all the nodes should have attributes, otherwise, use node.attributes?.unset('data-latex') instead. The data-latex-item attributes should no longer appear (they were removed in v4.1.0), so there is no need to remove them.

[dpvc]

Again, node.walkTree() could be used here, but there is a better way. The TeX input jax keeps a list of all the different types of nodes that it creates, and you can use that to get all the mspace and mo elements, instead of having to go through the entire tree, if you use a TeX input jax post-filter rather than a render action. E.g., when you call mathjax.document(), use something like

  InputJax: new TeX({
    packages: argv.packages.split(/\s*,\s*/),
    postFilters: [
      ({data}) => {
        for (const kind of ['mo', 'mspace']) {
          for (const node of data.getList(kind)) {
            if (node.attributes.getExplicit('linebreak') === 'newline') {
              node.attributes.unset('linebreak');
            }
          }
        }
      }
    ]
  });

Of course, you cold make the filter into a separate function if you like.

[dpvc]

Note that in v4, output rendering can require loading font character ranges dynamically. Usually, that is an asynchronous action, and you would need to use a promise-based approach, here, which would make it difficult to do this within a renderAction, which must be synchronous.

MathJax v4.1.3, however, can load font ranges synchronously, provided you are using a synchronous loader. Fortunately, the util/asyncLoad/node.js you are using is a synchronous loader, and so this will work, here, in v4.1.3 and later.

[dpvc]

Technically, setupEngine() may be asynchronous, and you should wait for Sre.engineReady() before calling other SRE functions. But a renderAction must be synchronous, so can't wait for a promise (I have a proposal to change this in v4.2). It may be that you can get away with treating this as synchronous, as you are here (I haven't tested it), since you have already loaded the locale earlier, but that is relying on undocumented behavior that could change in the future.

The same goes for the speech and Braille actions below.

[dpvc]

There is now html.renderPromise() that you can use directly rather than needing to use mathjax.handlRetriesFor() yourself.

[dpvc]

Since you mention the problem arising from < in the data-latex attributes (which is valid in HTML but not XML), you probably want to use adaptor.serilalizeXML() here rather than adaptor.outerHTML(). That will produce proper XML attribute values.

[rbeezer]

@dpvc - thanks again for the careful comments and suggestions. Just what I needed, and the insights for upcoming features and changes are a big help. No need to review the following, I think this is in good shape.

Force-push. Script bug has been handled separately, all changes have been consolidated into one commit. Comment coming with individual changes described.

I'll plan to let this sit for a few more days before merging, in case there are anymore comments from knowledgeable developers on either the PreTeXt side or the MathJax side.

[rbeezer]

Davide — thank you for the careful review. All eight suggestions were worked through, and each change was verified against the prior state of the branch: all four representation formats (mml, svg, speech, nemeth) of our sample article (670 items) are byte-identical throughout. The branch has been rebased onto current master and remains a single commit; the -c math repair moved to its own PR (#3011, merged).

Package list from source.js. Adopted, with a Set for the exclusions. Each configuration module is located and loaded directly, the same way cjs/util/Tex.js in MathJax-demos-node does; base is added explicitly since it is not a component. The resulting list is identical to what the registry enumeration produced. We considered the mixed setup for dynamically-loaded components and prefer preloading everything: the script is a deterministic batch job, and \require is not author-facing in PreTeXt.

MmlVisitor and the data-latex attributes. Adopted in full. The internal tree is no longer touched, so the attributes reach the Speech Rule Engine, including in the serialized MathML it is handed for speech and braille. MathML destined for output goes through the menu's MmlVisitor (its default filterTex/filterSRE options), and the SVG output uses CommonWrapper.skipAttributes['data-latex'] = true. Zero data-latex in any output, and a raw < (as in \(a < b\)) probed through every mode parses as XML, with correct speech.

Tree walker, data-latex-item. Both hand-rolled walkers are gone: the data-latex scrub is dropped entirely (above), and the newline scrub uses node.walkTree() with the node.attributes?.unset() guard. Good to know data-latex-item disappeared in 4.1.0.

Newline scrub as a TeX post-filter. Considered hard, and deliberately kept inside the SVG render action: the one TeX parse is shared by every output, and when SVG is combined with other outputs in a single run, the MathML, speech, and braille must keep the author's newline — only the SVG ignores it, as version 3 did. (PreTeXt invokes the script once per format, but the command line accepts combinations, and a combined --speech --braille run was verified to match the single-format runs.) A comment records the constraint.

Synchronous font-range loading. Good to know — a comment now records it next to the asyncLoad/node.js require, and package.json already carries ^4.1.3 as the floor.

setupEngine() and engineReady(). Restructured to stay on the documented path: every requested engine configuration is applied up front, each awaited through engineReady(), before rendering begins. In PreTeXt's actual usage — one output format per run — the render actions now make no setupEngine() calls at all; only a run combining several engine-consuming outputs re-applies configurations per pass, the synchronous-switch reliance you describe, now confined to that case and commented. An asynchronous render action in 4.2 would let us drop even that.

renderPromise(). Adopted.

serializeXML(). We tried it, and it cannot be adopted as-is: it adds xmlns="http://www.w3.org/1999/xhtml" to the bare html root (the lite parser's XMLNS map), which moves the whole mock page into the XHTML namespace, and every namespace-less element match in our packaging stylesheet then misses — the packaged file loses all of its structure. The attribute escaping is the half that mattered, and that hazard is now gone at the source: with data-latex filtered from every output, no output attribute can hold a raw < (and the MathML handed to the Speech Rule Engine is escaped by quoteHTML). A comment at the output call records the reasoning.

Claude Fable 5, acting as a coding assistant for Rob Beezer