Introduce a "file" element to supersede "datafile"

[skiadas]

Summary

New file element as successor to datafile; @format (pre/image/binary) names the kind, @user-interaction (none/view/edit) the visibility. datafile stays valid but is deprecated and auto-converted in the pre-processor (@editable="yes"→edit, @hide="yes"→none, else view). Adds first-class binary support.

Runestone integration

The PreTeXt side is complete; the Runestone side implements it:

• Binary delivery: HTML carries binary as a hidden base64 payload flagged data-isbinary (+ data-mime-type). The Runestone datafile component reads it, and server-side (Jobe) programs receive the real file in their working directory via @add-files / @compile-also — the base64 round-trips byte-for-byte (the sample book's gzip example verifies this).
• @compile-also is per-language. "Add to compilation" means different things depending on the program's @language and whether the file is text or binary:
  • text source files are compiled together with the main program (the traditional meaning);
  • a compiled Java/Kotlin archive (.jar/.zip) is placed on the compiler's classpath (-cp), so its classes are visible to the program;
  • a compiled C/C++ object file or static archive (.o/.a) is placed on the link line, after the source, so it is linked into the executable;
  • for other (interpreted) languages the file is simply available in the working directory.
  • @compile-also also folds the file into the program's @add-files, so authors only need to name it once.
• Known limitations:
  • Binary C/C++ shared libraries (.so) are not supported: the server cannot arrange for the runtime linker to find them.
  • In-browser runtimes (e.g. Skulpt for Python, GDScript) cannot consume binary files; a binary file is dropped from static (non-Runestone) output.
  • user-interaction="upload" (reader uploads a file into a program) needs Runestone server per-reader persistence; a one-line schema value reintroduces it later.
  • Unchanged: editable text (data-edit, textarea) and hidden files (user-interaction="none" — registered but not displayed) behave as before.

Related

• #2755 — deprecate datafile/pre/@source
• #2756 — earlier datafile deprecation PR (predecessor to this work)

[ascholerChemeketa]

@skiadas - didn't do an exhaustive test, but tried it out and also verified that it doesn't appear to change output for existing items. That looks great.

Suggestions (don't take these as gospel):

• Might be easier to verify there is no harm to old content if the new examples for SB came after the repair for old syntax. That would make it easier to check out a commit midway, then build and diff the output to verify old syntax still produces the same output.
• After the rewrite commit it might be nice to replace all <datafile>s in SB with the new syntax. Old syntax lurking in the defacto documentation means people don't tend to notice there is a newer way of doing things.
• The earlier commits should be reworded to match the naming style used in later ones.
• For the schema, you only should PR the pretext.xml file ... see the README.md in the schema folder.

[skiadas]

@skiadas - didn't do an exhaustive test, but tried it out and also verified that it doesn't appear to change output for existing items. That looks great.

Suggestions (don't take these as gospel):

• Might be easier to verify there is no harm to old content if the new examples for SB came after the repair for old syntax. That would make it easier to check out a commit midway, then build and diff the output to verify old syntax still produces the same output.
• After the rewrite commit it might be nice to replace all <datafile>s in SB with the new syntax. Old syntax lurking in the defacto documentation means people don't tend to notice there is a newer way of doing things.
• The earlier commits should be reworded to match the naming style used in later ones.
• For the schema, you only should PR the pretext.xml file ... see the README.md in the schema folder.

Thanks @ascholerChemeketa ! I updated the PR following those. I think I addressed all the points.

Two things I want to emphasize:

1. There are some whitespace diffs in the search index as a result of the presence or absence of a pre element between the before- and after- versions.
2. Because of an expected invariant on the assembly-id computations, the repair step isn't allowed to replace one element with two. As a result there is another slight drift between the version before the example conversion and the one after. The general recommendation for datafile images is to instead do a file tag with hide interaction, but then add a sibling image tag pointing to the same source. The new example files that replaced the old example datafiles do that. However the repair script isn't allowed to turn the single datafile entry into a pair of file+image entries. As a result the repair script instead produces file tags that contain images but are also set to view, something which in general should be discouraged in favor of the 2 tag approach. This results in whitespace diffs again. But it was the cleanest way I could see to do the repair while maintaining the invariant.