Skip to content

docs: migrate documentation toolchain from asciidoc-py + dblatex to asciidoctor#4053

Merged
BsAtHome merged 10 commits into
LinuxCNC:masterfrom
grandixximo:docs/asciidoctor-migration
May 30, 2026
Merged

docs: migrate documentation toolchain from asciidoc-py + dblatex to asciidoctor#4053
BsAtHome merged 10 commits into
LinuxCNC:masterfrom
grandixximo:docs/asciidoctor-migration

Conversation

@grandixximo
Copy link
Copy Markdown
Contributor

@grandixximo grandixximo commented May 24, 2026

Summary

Replaces the docs toolchain: asciidoc-py + dblatex + xsltproc + source-highlight + inkscape are dropped, build now goes through asciidoctor + asciidoctor-pdf + rouge. Motivation as discussed on #4051: asciidoc-py is EOL, dblatex unmaintained, asciidoctor is actively packaged in Debian and uses prawn-svg natively (no inkscape detour). Continues hansu's earlier asciidoctor branch and solves the cross-document anchor problem that stalled it.

What changed (10 commits, oldest first)

  1. docs: add Rouge syntax highlighters for HAL, NGC, INI — three custom Rouge lexers (rouge_hal/ngc/ini.rb) loaded by asciidoctor via -r. NGC handles LinuxCNC #<_ini[...]NAME> / #<_hal[...]> references.

  2. docs: add asciidoctor extensions for image lookup and cross-doc xrefsimage_resolver.rb (treeprocessor: per-lang tree, EN fallback, foo_en.ext -> foo_<lang>.ext substitution, default pdfwidth=75%); xref_resolver.rb (preprocessor: bare <<anchor,Title>> -> qualified <<relpath/file.adoc#anchor,Title>>, anchor index cached on disk by mtime).

  3. docs: HTML/PDF theme CSS overrides and docinfolcnc-overrides.css (heading #1a3a6c incl. #header h1, prefers-color-scheme dark, no JS), pdf-theme.yml (asciidoctor-pdf theme: A4, dblatex-blue headings, top header doc-title | chapter | page/total), docinfo.html, static-page CSS refresh.

  4. docs: portable CJK font (CFF -> TTF) converterotf2ttf.py builds a TrueType (glyf) copy of Noto Serif CJK SC from the system .ttc. prawn 2.4 corrupts CFF embeds. Subsets to glyphs actually used in docs + .po catalogues (~1.5s vs ~200s for full face).

  5. build: probe optional docs deps; graceful disable, hard-fail explicitconfigure.ac probes ruby-rouge, python3-fonttools, NotoSerifCJK fontconfig. Warn-and-disable when implicit; hard-fail for explicit --enable-build-documentation=pdf. debian/configure + control.top.in swap build-deps to asciidoctor stack.

  6. docs: migrate doc build rules to asciidoctor toolchain — rewrites Submakefile rules around asciidoctor + asciidoctor-pdf + Rouge. Idempotent stamp-gated build, per-lang target lists derived from po4a.cfg, manpage HTML from adoc, master PDF assembled from PDF_MAN_ORDER. Drops legacy infra: xhtml11*.conf, docbook*.conf, emc2.sty, html-images.xslt, links.xslt, links_db_gen.py, source-highlight/, scripts/inkscape.

  7. docs: source content fixes for asciidoctor parser — adjustments where asciidoctor is stricter than asciidoc-py: table cells count (halmodule, qtplasmac), missing block delimiters (qtdragon), fenced code blocks (qtvcp-widgets), Inkscape flowRoot in Spanish lathe SVG, blank lines before * bullets, plus INI/HAL/NGC source cleanup the new lexers exposed.

  8. docs: rewrite gen_complist.py + po4a.cfg for per-language aliases — content-stable mtime via write_if_changed. Bertho identified that components_gen.adoc was rewritten with HTML-existence-dependent content every build, bumping documentation.pot mtime and making po4a look like it ran twice.

  9. build: misc hygiene (gitignore, autom4te.cache, manpage images) — drop nocheck workaround, ignore autom4te.cache + .checklink.*.tmp, per-lang objects/.fonts ignore.

  10. ci: enable translated docs build in build-doc.sh — passes --enable-build-documentation-translation so CI exercises the per-language pipeline.

Output verification

dblatex (official 2.9) asciidoctor (this branch)
Master PDF 26 MB, 1347 pages 27 MB, 1440 pages
33-PDF total 243 MB 238 MB
HTML files ~1551 1551
Manpages 1287 1287
make -j8 pdfdocs wall time 4m56s 3m43s
Full Debian binary-indep (parallel) n/a 7 min
Build warnings n/a 0
Idempotent (2nd run no-op) no yes

CI: htmldocs ~13m vs ~19m on master; package-indep ~16-19m vs ~22-23m. Spot-checked sample pages render correctly across en, de, ru, uk, zh_CN.

Open items (follow-up PRs, agreed with reviewers)

  • Unified output subtree docs/build/{html,pdf,man}/<lang>/.
  • Translated manpages built from adoc (one of the follow-up PRs already prototyped at test: ui-smoke phase 2, g-code execution and endpoint check #4054).
  • Topbar + per-page language switcher with grey-out for missing translations.
  • pot/po split (manpages and GUI strings out of documentation.pot).

PDF TOC trim, lingering INI source content cleanups, more CSS slimming: tracked but not blocking.

Test plan

  • make -j8 pdfdocs clean, all 33 PDFs
  • make htmldocs clean, 1551 files
  • make manpages clean, 1287 pages
  • fakeroot debian/rules binary-indep produces linuxcnc-doc-en + linuxcnc-doc-de
  • Second consecutive build is a no-op (idempotent)
  • Cyrillic in code blocks (uk); CJK in titles (zh_CN)
  • All build deps present on bookworm, trixie, sid, noble

cc @hansu, @BsAtHome

@grandixximo grandixximo force-pushed the docs/asciidoctor-migration branch 4 times, most recently from cc516a5 to 5b775f1 Compare May 24, 2026 02:46
@grandixximo
Copy link
Copy Markdown
Contributor Author

Tangential, but tied to docs UX: a few months back in one of the Sunday maintainer meetings we discussed adding navigation aids to the HTML docs. A "back to index" link from each page, and a top bar with a few quick links. I'm not sure if anyone has taken that up since (I checked @smoe's fork branches and didn't find anything matching, but I may have missed it).

A sidebar Table of Contents wasn't part of that conversation as far as I remember, but it feels like a natural fit alongside the rest, the current top-of-page TOC gets quite long.

If it's still on the wish list, I'd be happy to do a follow-up PR after this one lands. The asciidoctor toolchain makes it cheap: -a toc=left gets the sidebar TOC, and -a docinfo=shared (which the new rule already enables) lets a small docinfo-header.html inject a top nav bar without touching any .adoc source. Happy to scope and propose first if anyone (cc @hansu) wants to chime in on what should go in the top bar.

@BsAtHome
Copy link
Copy Markdown
Contributor

The man-page translations take the wrong source from the generated troff files. There were originally only troff files, but the manpages are now in asciidoc format under the docs/src/man/* tree. Except for the component generated asciidoc pages that are generated in src/object/man/*. This should also be fixed and especially the HTML manpages must be generated from the adoc sources.

There has been, for a long time at least on my system, a bug in the docs build that running make twice was required to build everything correctly. At least, the second invocation was not silent and actually made stuff.

You changed the highlighter. Does it support NGC and INI? How are you highlighting HAL files, which is a LinuxCNC specific format? The highlight format filesfor these three are added "manually" in the current build (with some effort).

There are at least two things on my wish list when building the docs:

  • Do not build translations or invoke any process that involves translation setup, including .pot/.po generation, unless explicitly requested. It also needs to be configured to enable running any process involved in translations.
  • Move all generated documents and translations, including the source language, into a subtree .../docs/build/{en,de,...}/{man,pdf,html,...}. Then everything generated is found in one place instead of all over the place.

grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
Bertho noted in the review of LinuxCNC#4053 that the new build dropped syntax
highlighting for HAL and NGC source blocks; rouge ships an INI lexer
but has neither of the two LinuxCNC-specific languages, so blocks like
[source,{hal}] and [source,{ngc}] rendered as plain text.

* Two rouge lexers, ~80 lines each, ported line-for-line from the old
  source-highlight definitions at docs/src/source-highlight/hal.lang
  and ngc.lang (Michael Haberler, 2011).  Same keyword coverage:
  halcmd commands, pin/signal names, INI substitutions and env vars
  for HAL; G/M/T/F/S codes, axis letters, parameters, O-words and the
  math/boolean built-ins for NGC.
* All four asciidoctor invocations in the Submakefile (PDF, HTML,
  manpage HTML, ASCIIDOCTOR_HTML_RULE) gain '-r .../rouge_hal.rb -r
  .../rouge_ngc.rb' so the lexers are visible to rouge before the
  document is parsed.  The manpage HTML rule also gains an explicit
  '-a source-highlighter=rouge' that the others already inherit from
  attribute defaults.
* The :ngc: / :hal: / :ini: / :css: / :nml: attribute defs in the
  source files used asciidoc-py's '{basebackend@docbook:'':ngc}'
  conditional syntax (which asciidoctor does not implement) to emit
  the language only when targeting docbook.  All toolchain backends
  used by this PR now want the language name unconditionally, so the
  attribute defs collapse to ':ngc: ngc' etc.  84 source files
  touched, no .adoc body changes.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
…-twice

Bertho noted in the review of LinuxCNC#4053 that:

  * a clean 'make pdfdocs' or 'make htmldocs' required a second pass
    to finish, because po4a generated the per-language .adoc files
    *during* the build but the make-time $(wildcard $(L)/*.adoc)
    expansion had already evaluated them as missing;
  * po4a should not run, and translation setup should not be invoked,
    on a developer build unless the developer asks for it.

Address both:

* configure.ac flips the default: BUILD_DOCS_TRANSLATED now requires
  an explicit '--enable-build-documentation-translation' (the old
  '--disable-build-documentation-translation' opt-out is replaced).
  Stale dblatex-era version probes and warnings around po4a are also
  removed; po4a >= 0.67 is required when the flag is on, missing or
  too-old po4a now errors instead of warning-and-disabling.

* debian/rules.in keeps the .deb pipeline producing translations by
  always passing '--enable-build-documentation-translation' alongside
  the existing '--enable-build-documentation=pdf'.

* docs/src/Submakefile:
  - Translated DOC_SRCS_<lang> are now derived from the AsciiDoc_def
    lines in po4a.cfg instead of $(wildcard $(L)/*.adoc).  The list
    is therefore correct on a fresh tree (po4a has not run yet) and
    no longer includes English-only sources like
    drivers/mesa_modbus.adoc that the translation pipeline does not
    touch.
  - DOC_SRCS and PDF_TARGETS only pull in the per-language lists when
    BUILD_DOCS_TRANSLATED=yes, so a default-configured build builds
    English only and never invokes po4a.
  - The orphaned 'xetex available?' check is dropped: prawn-svg in
    asciidoctor-pdf renders CJK from our TTF subset, xetex is no
    longer a build-time gate.
  - When BUILD_DOCS_TRANSLATED=yes, an empty-recipe pattern rule
    associates every translated .adoc with translateddocs as an
    order-only prerequisite, so 'make pdfdocs' (or 'make htmldocs')
    on a clean tree triggers po4a before depends/%.d evaluation,
    eliminating the two-pass requirement.
@BsAtHome
Copy link
Copy Markdown
Contributor

For translated images,...
We need to have a standard naming convention for all image names. The default image would then be the one with the img_en.ext name (the English version). If there are translated images, then they are named as such in the source tree (img_de.ext, img_es.ext, ...). Images are a special case and generally cannot be auto-translated.

@grandixximo
Copy link
Copy Markdown
Contributor Author

Thanks Bertho. Pushed two commits (b0a16fc, 7f511e8):

HAL / NGC highlighting: rouge HAL and NGC lexers, ported line-for-line from the old docs/src/source-highlight/hal.lang and ngc.lang, same keyword coverage as before. INI was already in rouge. The :ngc:/:hal:/:ini: attribute defs collapsed to plain :ngc: ngc (asciidoc-py's {basebackend@docbook:...} conditional is not implemented by asciidoctor).

Build twice: reproduced and fixed. Cause was $(wildcard $(L)/*.adoc) evaluating at make parse time before po4a generated the files. Now reads the translated-file list straight from po4a.cfg, plus an empty-recipe order-only rule so make pdfdocs on a fresh tree triggers po4a first.

Translation opt-in: --enable-build-documentation-translation (default off) replaces the old --disable-… opt-out. make pdfdocs on a default configure builds English only and never invokes po4a. debian/rules.in keeps passing the flag so the .deb builds still produce all languages.

Manpage HTML from troff: I think this is a misread, the recipe at docs/src/Submakefile:498-541 reads from .adoc, the troff dep is just so the rule can detect troff-level .so aliases (iocontrol.1 -> io.1) and symlink them in HTML.

docs/build/{lang}/{man,pdf,html} subtree: agreed it's the right shape. It's a sizeable touch, docs/index.html (auto-generated) and the linuxcnc.org public download URLs point at docs/LinuxCNC_*.pdf directly, so the move needs redirect symlinks or coordination with the website deploy. Should this land as part of this PR? cc @hansu @andypugh @smoe for thoughts.

Comment thread docs/src/code/code-notes.adoc Outdated
Comment thread docs/src/extensions/rouge_hal.rb
Comment thread docs/src/extensions/rouge_hal.rb Outdated
Comment thread docs/src/extensions/rouge_hal.rb Outdated
Comment thread docs/src/extensions/rouge_hal.rb Outdated
Comment thread docs/src/extensions/rouge_ngc.rb
@hansu
Copy link
Copy Markdown
Member

hansu commented May 24, 2026

Thanks for resuming the work on this!

While trying to install the dependencies I wonder why the configure (./configure --with-realtime=uspace --enable-build-documentation=pdf,html) succeeds and only prints warnings about packages needed for building the docs:

checking for asciidoctor... none
configure: WARNING: no asciidoctor, documentation cannot be built
checking for asciidoctor-pdf... none
configure: WARNING: no asciidoctor-pdf, PDF documentation cannot be built
...
checking for rsvg-convert... none
configure: WARNING: no rsvg-convert, documentation cannot be built

Further the build failed when the font NotoSerifCJK-Regular.ttc was needed and I couldn't find the dependency for that, so I installed fonts-noto-cjk.

@BsAtHome
Copy link
Copy Markdown
Contributor

Manpage HTML from troff: I think this is a misread, the recipe at docs/src/Submakefile:498-541 reads from .adoc, the troff dep is just so the rule can detect troff-level .so aliases (iocontrol.1 -> io.1) and symlink them in HTML.

The problem was in the PDF generation. It used the troff files as input and doing so could no longer syntax highlight code snippets.

Secondly, how do the components' man pages get involved here? They are not in the $(DOC_DIR)/man/% place afaik.

@hansu
Copy link
Copy Markdown
Member

hansu commented May 24, 2026

You changed the highlighter. Does it support NGC and INI? How are you highlighting HAL files, which is a LinuxCNC specific format? The highlight format filesfor these three are added "manually" in the current build (with some effort).

Currently the syntax-hightlighting for both is gone. Why do you had to switch to rouge?

Comment thread docs/src/Submakefile Outdated
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
hansu pointed out on LinuxCNC#4053 that

  ./configure --with-realtime=uspace --enable-build-documentation=pdf,html

happily succeeds with only WARN-level diagnostics when asciidoctor /
asciidoctor-pdf / rsvg-convert are absent, silently flipping BUILD_DOCS
back to "no".  Running 'make pdfdocs' afterwards produces no docs and
no clear hint that the configure step had stripped the docs targets.

Convert all of those AC_MSG_WARN+disable paths to AC_MSG_ERROR with an
'apt-get install ...' hint.  Same treatment for ghostscript (the PDF
post-process), librsvg2-bin (SVG -> PDF/PNG) and w3c-linkchecker for
the HTML side.

Also add an AC_MSG_ERROR for the NotoSerifCJK font when PDF docs are
enabled.  The Submakefile depends on the .ttc unconditionally (the
CJK glyph fallback is wired into every PDF, not only the translated
ones), so missing fonts-noto-cjk used to surface as a cryptic
'No rule to make target NotoSerifCJK-Regular.ttc' at build time.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
Bertho noted on LinuxCNC#4053 that hardcoding the .ttc paths under
/usr/share/fonts/opentype/noto/ pins the build to Debian / Ubuntu
and will break Arch, Fedora, openSUSE, etc.

Move the discovery into configure.ac.  It first asks fontconfig
(`fc-match --format='%{file}' 'Noto Serif CJK SC:style=...'`) and
falls back to the package paths that the major distributions
actually use (Debian, Arch noto-fonts-cjk, Fedora
google-noto-cjk-fonts).  The probe rejects anything that is not a
.ttc, because otf2ttf.py needs the TrueType Collection to pick
index 2 (SC) out of it.  If nothing matches, configure errors with
a per-distro install hint, and the user can override with
   ./configure NOTOCJK_REGULAR_TTC=/path/to/Regular.ttc \
              NOTOCJK_BOLD_TTC=/path/to/Bold.ttc

The resolved paths flow through Makefile.inc as NOTOCJK_REGULAR_TTC
and NOTOCJK_BOLD_TTC; docs/src/Submakefile now references the
substituted variables instead of the literal /usr/share path.
@grandixximo
Copy link
Copy Markdown
Contributor Author

You changed the highlighter. Does it support NGC and INI? How are you highlighting HAL files, which is a LinuxCNC specific format? The highlight format filesfor these three are added "manually" in the current build (with some effort).

Currently the syntax-hightlighting for both is gone. Why do you had to switch to rouge?

old syntax-hightlighting is not supported in asciidoctor that's why I had to switch.

@BsAtHome
Copy link
Copy Markdown
Contributor

Building arch package:

checking whether to build documentation... PDF requested
checking for asciidoctor... /usr/bin/asciidoctor
checking for gs... none
configure: error: no gs, cannot build documentation
install with "sudo apt-get install ghostscript"
...

Why does it say "PDF requested" in an arch package?

grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
Bertho reported on LinuxCNC#4053 that building an arch-only Debian package
(debuild --build=any) now errors at configure time with

    checking whether to build documentation... PDF requested
    checking for gs... none
    configure: error: no gs, cannot build documentation

The root cause is f314b5c.  debian/rules.in passes
--enable-build-documentation=pdf unconditionally, but
--build=any deliberately skips Build-Depends-Indep (the docs
toolchain lives there), so the configure-time docs probes never had
gs / asciidoctor / rsvg-convert available on that path.  The
previous WARN-and-disable behaviour quietly turned BUILD_DOCS off
and the arch-only build completed; the new hard error breaks it.

Restore WARN+disable for asciidoctor / gs / rsvg-convert /
asciidoctor-pdf / checklink, but keep the new hard error for the
NotoSerifCJK font.  The font check was the actual hansu complaint:
missing fonts-noto-cjk used to surface as a cryptic
"No rule to make target NotoSerifCJK-Regular.ttc" deep in the make
log, and a clear configure error there is worth the strictness.
Skipping that check when BUILD_DOCS_PDF is already off keeps it
out of the arch-only path.
@grandixximo
Copy link
Copy Markdown
Contributor Author

Building arch package:

checking whether to build documentation... PDF requested
checking for asciidoctor... /usr/bin/asciidoctor
checking for gs... none
configure: error: no gs, cannot build documentation
install with "sudo apt-get install ghostscript"
...

Why does it say "PDF requested" in an arch package?

I changed to errors, but should have stay as warning try 3c30d24

@grandixximo
Copy link
Copy Markdown
Contributor Author

Manpage HTML from troff: I think this is a misread, the recipe at docs/src/Submakefile:498-541 reads from .adoc, the troff dep is just so the rule can detect troff-level .so aliases (iocontrol.1 -> io.1) and symlink them in HTML.

The problem was in the PDF generation. It used the troff files as input and doing so could no longer syntax highlight code snippets.

Secondly, how do the components' man pages get involved here? They are not in the $(DOC_DIR)/man/% place afaik.

Manpage PDF: rewritten in 95f9d0c to go through asciidoctor-pdf from the .adoc sources, so code blocks keep their highlighting. A small master document is generated each build that includes every entry in PDF_MAN_ORDER as a chapter (leveloffset=+1) with a hard page break between entries.

Component manpages: halcompile drops the troff .N into $(DOC_DIR)/man/manN/ and the matching .adoc into objects/man/manN/ (src/hal/components/Submakefile:57-62). The new manpage PDF rule checks $(DOC_SRCDIR)/man/$$M.adoc first, falls back to objects/man/$$M.adoc, same lookup the HTML rule already uses at docs/src/Submakefile:536-543, so component pages are picked up alongside the native ones.

grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
Bertho noted on LinuxCNC#4053 that the LinuxCNC system parameters
#<_ini[SECTION]NAME> (read a value from the INI file) and
#<_hal[pin.or.signal]> (read a HAL pin / signal) are dialect-specific
enough that they deserve to stand out from the rest of variable
expansion in code samples, rather than being lumped under the
generic #<...> named-parameter rule.

Add two dedicated rules ahead of the generic one and tokenise the
sub-parts separately so the section / pin name reads differently:

  #<_ini[SECTION]NAME>
   ^   ^   ^      ^    ^
   |   |   |      |    Name::Variable
   |   |   |      Name::Constant
   |   |   Name::Constant + Punctuation brackets
   |   Name::Builtin
   Name::Variable

  #<_hal[pin.or.signal]>
   ^   ^   ^             ^
   |   |   |             Name::Variable
   |   |   Name::Attribute (same token as a HAL pin elsewhere)
   |   Name::Builtin
   Name::Variable

Generic #<myvar> and LinuxCNC#100 fall through to the existing rules.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 24, 2026
Bertho proposed on LinuxCNC#4053 a uniform naming convention for translated
images: the default is img_en.ext (English), translated variants
live alongside it as img_de.ext, img_fr.ext, ... in the same source
directory.  The author writes img_en.ext in the .adoc; the build
silently picks the localised file when it exists.

The source tree already follows the convention partially (see
docs/src/gcode/images/comp-path_fr.svg next to comp-path_en.svg).
What was missing is the lookup.  Extend the asciidoctor image
resolver:

  - infer the document language from the .adoc's path
    (docs/src/<lang>/... -> <lang>, otherwise 'en')
  - if the macro target ends with _en.<ext> and the document
    language is not en, try the _<lang>.<ext> variant first; fall
    back to the original target if the translation does not exist
  - applied to both block (image::) and inline (image:) macros
  - PDF backend still rewrites to absolute paths; HTML now also
    receives the relative-target rewrite when a translated variant
    is picked, so .html-images-stamp copies the right file

English-only docs and macros without an _en suffix are untouched.
@grandixximo
Copy link
Copy Markdown
Contributor Author

For translated images,...
We need to have a standard naming convention for all image names. The default image would then be the one with the img_en.ext name (the English version). If there are translated images, then they are named as such in the source tree (img_de.ext, img_es.ext, ...). Images are a special case and generally cannot be auto-translated.

Implemented in 2c5075c. The asciidoctor image resolver now infers the doc language from the source path (docs/src/<lang>/...), and when a macro target ends with _en.<ext> it tries the _<lang>.<ext> sibling in the same directory first, falling back to _en if the translated variant is not in the tree. Applies to both block image:: and inline image: macros, on PDF and HTML. Authors keep writing img_en.ext in the source; translators only need to drop img_<lang>.ext next to it when a localised variant exists. English-only docs and macros without an _en suffix are untouched.

@hansu
Copy link
Copy Markdown
Member

hansu commented May 24, 2026

Some little styling things I noticed:

  • The colors for the syntax-highlighting need some adjustment
  • the left margin is very small
  • the header font is now one with serif
  • the whole line of a TOC item is clickable

(left: new, right old)

grafik

@BsAtHome
Copy link
Copy Markdown
Contributor

Some little styling things I noticed:

  • The colors for the syntax-highlighting need some adjustment

That should be set as a "style" argument in calling rouge. Maybe a possible cmd-line override for people wanting different color styles?

  • the left margin is very small

Indeed. There needs to be done some work to scale properly on different size/width screens. Maybe some media breakpoints need to be added to the css.

  • the header font is now one with serif

Yes, that does look strange. ;-)

  • the whole line of a TOC item is clickable

Wasn't that always the case?

The header lines (blueish background) in the old version lack left padding. That is fixed in the new version.

The "Note" formatting (asciidoc's admonitions) needs some more work. The "Asciidoctor.js Live Preview" I use in Firefox uses proper icons, which are much more visible than text only. Maybe they can be styled into icons?

I do think that the "blueish" header lines are not looking nice. When we're done with the build issues, maybe we should take a general look at the css and do some updates.

@hansu
Copy link
Copy Markdown
Member

hansu commented May 24, 2026

Some little styling things I noticed:

  • The colors for the syntax-highlighting need some adjustment

That should be set as a "style" argument in calling rouge. Maybe a possible cmd-line override for people wanting different color styles?

Yes this can be passed to the call of asciidoctor. I tried some, but didn't manage yet to find a suitable one.

  • the left margin is very small

Indeed. There needs to be done some work to scale properly on different size/width screens. Maybe some media breakpoints need to be added to the css.

The default asciidoctor output without passing any css looks much better regarding this.
But I don't have time now to investigate this further. Just wanted to add it here that it doesn't get forgotten.

  • the header font is now one with serif

Yes, that does look strange. ;-)

  • the whole line of a TOC item is clickable

Wasn't that always the case?

No. It feels weird when you click on the white background and end up somewhere in the middle.of the document.

The header lines (blueish background) in the old version lack left padding. That is fixed in the new version.

The "Note" formatting (asciidoc's admonitions) needs some more work. The "Asciidoctor.js Live Preview" I use in Firefox uses proper icons, which are much more visible than text only. Maybe they can be styled into icons?

I do think that the "blueish" header lines are not looking nice. When we're done with the build issues, maybe we should take a general look at the css and do some updates.

I think that's kind of a part of LinuxCNC's corpoate design ;-)

@BsAtHome
Copy link
Copy Markdown
Contributor

I do think that the "blueish" header lines are not looking nice. When we're done with the build issues, maybe we should take a general look at the css and do some updates.

I think that's kind of a part of LinuxCNC's corpoate design ;-)

The pain receptors in my eyes would be forever grateful if we could leave the early designs of the web behind and agree that some things are best left to history ;-)

Anything more modern and absolutely, definitely, never ever requiring Javascriptthe destroyer is very much appreciated.

@grandixximo
Copy link
Copy Markdown
Contributor Author

grandixximo commented May 24, 2026

"blueish" header lines are not looking nice.

they help with contrast when using Dark Reader, we don't have a dark theme, least we can do is play nice with dark theme extensions.

@BsAtHome I'm not sure why CI is taking hours to complete, ideas?

@hansu
Copy link
Copy Markdown
Member

hansu commented May 29, 2026

1. Unified output subtree docs/build/<lang>/{html,man-html,man,pdf}/

* Layout A (symmetric): `docs/build/en/`, `docs/build/de/`, etc. Every language equal.

* Layout B (en flat, langs nested): `docs/build/{html,pdf,man}/` for English at root, `docs/build/<lang>/...` for translations.

I prefer a symmetrical layout but would go without the additional build folder, so like
docs/{html,pdf,html/man}<lang>/
That would be consistent with the access to the website like: https://linuxcnc.org/docs/devel/html/de/gui/gmoccapy.html

And hope that we can also get away from the files that are copied during build to docs/src/<lang>.

2. Manpage HTML placement

* Nested: `docs/build/<lang>/html/man/manN/X.html` (single html tree per lang, current pattern).

This one!

4. pot/po split scope

You flagged the components_gen.adoc -> documentation.pot mtime chain as still drifting. Smallest split that fixes it:

* core + components (2-way), or

* core + components + manpages (3-way), or

* finer (per-Master, per-section)?

What do you want to split?

What gives translators the right logical units without too many catalogues to chase?

5. Stack order, one PR, 4 commits: ```

The stack order of the commits of the follow-up PR?

@hansu
Copy link
Copy Markdown
Member

hansu commented May 29, 2026

The header color should be also in dark blue like the section headings:

#header>h1:first-child{
   color: #1a3a6c;
}

@grandixximo
Copy link
Copy Markdown
Contributor Author

@hansu thanks. Follow-ups:

1. Layout: your docs/{html,pdf,man}/<lang>/ mirrors the served URL 1:1. Trade-off:

  • Yours: deploy rsync unchanged, no infrastructure PR. Per-lang clean = find docs/html/<L> docs/pdf/<L> docs/man/<L> -delete.
  • Mine docs/build/<lang>/...: per-lang clean = rm -rf docs/build/<L>/. Needs deploy translation.

Yours is less work for less reward. Going with yours unless @BsAtHome disagrees.

Follow-up question: translated .adoc files. po4a currently writes to docs/src/<lang>/ polluting the source tree. Move to docs/objects/<lang>/ (sibling of src/objects/) or docs/intermediate/<lang>/? Need a non-src home.

2. Manpage HTML nested: agreed. docs/html/<lang>/man/manN/X.html.

4. pot/po split (clarifying): @BsAtHome's complaint: docs/src/hal/components_gen.adoc is generated each build; documentation.pot depends on it; mtime bump makes po4a look like it ran twice. His suggestion: pull component strings into a separate components.pot so only that catalog re-runs.

My Q4 was about granularity:

  • 2-way: documentation.pot + components.pot. Smallest fix.
  • 3-way: documentation.pot + components.pot + manpages.pot. Cleaner logical units for translators.
  • finer: probably overkill.

Which?

5. Stack order (clarifying): 4 commits in one follow-up PR. Question is which lands first. Layout (commit 1) first = commits 2-4 written against final paths, easier review. Layout last = each per-commit smaller, but final mechanical sweep is big. Preference?

Header color: ack, #header>h1:first-child { color: #1a3a6c; }. Will push to docs/asciidoctor-migration as a small follow-up commit on top, separate from the path refactor.

@hansu
Copy link
Copy Markdown
Member

hansu commented May 29, 2026

Follow-up question: translated .adoc files. po4a currently writes to docs/src/<lang>/ polluting the source tree. Move to docs/objects/<lang>/ (sibling of src/objects/)

docs/objects/<lang>/ sounds good.

4. pot/po split (clarifying): @BsAtHome's complaint: docs/src/hal/components_gen.adoc is generated each build; documentation.pot depends on it; mtime bump makes po4a look like it ran twice. His suggestion: pull component strings into a separate components.pot so only that catalog re-runs.

My Q4 was about granularity:

* 2-way: `documentation.pot` + `components.pot`. Smallest fix.

I think 2-way would be sufficient. But this would require a new component on the Weblate side only for the components list AFAIK. Can't we configure the makefile that it only runs when man pages are changed?
Or we could run this generation optionally... On the CI this isn't an issue, right?

5. Stack order (clarifying): 4 commits in one follow-up PR. Question is which lands first.

No strong opinion about that.

But when you are talking about commit order - what do you guys think about bringing a little order to the commits? 69 now and some times they go a bit back and forth which makes it a bit difficult to trace the changes. But I guess the effort would be disproportionately high compared to the benefit.

@BsAtHome
Copy link
Copy Markdown
Contributor

Unified output subtree** docs/build/<lang>/{html,man-html,man,pdf}/

  • Layout A (symmetric): docs/build/en/, docs/build/de/, etc. Every language equal.

@hansu the idea is to isolate everything generated into one directory. No more "where did it get placed?" problems. The same is done for the main body of code that is compiled into src/objects/ directory.

A uniform structure is more consistent. Each language has its place. However, it does pose a problem in the consistency when using it in-tree locally. All html pages have an added indirection. It may therefore be prudent to take a different route so that we have both build, local and remote consistency:

docs/build/
    src/
        <lang>/
            ... # all docs translated per language
    pdf/ # *** see below ***
        <lang>/
            ...generated PDF documents...
    html/
        index.html # static page with redirect to lang=en
        css/ # all common css
        images/ # all common images
        <lang>/
            man[1,3,9]/ # html man-pages
            images/ # localized images
            pdf/ # *** see below ***
                ...generated PDF documents...
            [css/] # if we ever need localized css
            index.html
            ....htmldoc structure...
    man/
        <lang>/
            man[1,3,9]/ # troff man-pages (consistent with /usr/share/man/<lang>/manX)

That way we have the entire HTML tree contained and can be exported as a tarball/zip/etc.. The tricky part is that you want to have the PDF docs linked in the HTML tree if you build both. We can make the PDF-tree subordinate to the HTML tree and leave it empty if not generated. This allows to zip up everything web-based in one go and only has local relative links.
The PDFs can be placed at 1) docs/build/pdf/<lang>/*.pdf 2) docs/build/html/pdf/*_<lang>.pdf or 3) docs/build/html/<lang>/pdf/*_.pdf. If we want the PDFs to be accessible from the web, then they should be part of the html tree. The potential downside with PDFs is that the PDF files all have the same name and when you download one, you cannot see from the name what language it is in. Or you can make it flat, but that breaks the <lang> isolation structure.

The docs packages take the instructions from the install-doc-pdf rule, which can install them from any place with the lang tag appended into the usr/share/doc tree. So it is not that important where they end up in the build, as long as it is consistent.

2. Manpage HTML placement

See above

3. External URL and install path contract
Refactor is internal only. debian/linuxcnc-doc-.docs updated to point at new source paths, install destinations unchanged (/usr/share/doc/linuxcnc-doc-/html/...). Companion PR in linuxcnc/infrastructure updates emc2-doc-build rsync to flatten new tree to legacy docs/devel/{html,pdf}/... on the server. No public URL change, no broken bookmarks. OK?

The homepage issue is moot once you have a tree that is locally traversable with only relative links. You only need to rsync one directory.

4. pot/po split scope

This will be a separate discussion. Too much for now.

@hansu at least the manpages should be separated out of the main body. The documentation.pot file is 233k lines. It is probably also prudent to separate out the GUIs (like gmoccapy).

5. Stack order, one PR, 4 commits

Multiple PRs, please. These large infrastructure changes should be isolated in PRs, not commits of one PR. This PR is already at ~200 comments...

@grandixximo
Copy link
Copy Markdown
Contributor Author

I will squash this up tomorrow morning, thanks for the feedback

@hansu
Copy link
Copy Markdown
Member

hansu commented May 29, 2026

Even though docs/build/src/ looks a bit weird, but agree to have a common dir and no separate objects dir.

If we want the PDFs to be accessible from the web, then they should be part of the html tree.

If we keep this page https://linuxcnc.org/documents/ then it's not needed. I don't think wee need to link to the PDFs from within the docs.

It is probably also prudent to separate out the GUIs (like gmoccapy).

You are talking about the documentation of the GUIs?

I will squash this up tomorrow morning, thanks for the feedback

Thanks, but don't spent too much time on this.

@BsAtHome
Copy link
Copy Markdown
Contributor

Even though docs/build/src/ looks a bit weird, but agree to have a common dir and no separate objects dir.

Yeah, the src in the build directory are the (translated) adoc sources, which are generated. Sometimes names can be confusing. We could call it docs/build/adoc/ for clarity? Anyway, having it all combined in one build subdirectory location helps.

If we want the PDFs to be accessible from the web, then they should be part of the html tree.

If we keep this page https://linuxcnc.org/documents/ then it's not needed. I don't think wee need to link to the PDFs from within the docs.

That current PDF listing page is just a (unattractive) list of files that you have to read line-by-line to find what you want. We can do better than that. We use the same language division and buildup as @grandixximo showed a peek preview of. One style for all doc pages. A better integrated system and more consistent.

I'm also (nearly) sure (and hope) that a better and more attractive presentation will improve people's willingness to contribute translations of the huge mountain of texts.

We can and should keep the link to PDFs from the documents' link page, but the target will simply be an integral part of the new html tree.

It is probably also prudent to separate out the GUIs (like gmoccapy).

You are talking about the documentation of the GUIs?

No, I was talking about the translation of messages in the GUIs' code. Those that share the same underlying system share the translation, of course. Maybe it would already be enough just to take all (G)UIs out into one (G)UI translation pot (them we'd have gmoccapy.pot and f.ex. lcnc_ui.pot). See it as "front-end" and "back-end" translation.

Splitting up the documentation into translation groups could be a future issue, but that can first be decided when we revise the organisational and content structure to become a logically coherent story (with the FIXMEs and TODOs fixed). A setup easy for all to follow and find things in. That is a huge undertaking and is future talk.

please note the careful use of weasel-words here ;-)

@hansu
Copy link
Copy Markdown
Member

hansu commented May 29, 2026

Even though docs/build/src/ looks a bit weird, but agree to have a common dir and no separate objects dir.

Yeah, the src in the build directory are the (translated) adoc sources, which are generated. Sometimes names can be confusing. We could call it docs/build/adoc/ for clarity? Anyway, having it all combined in one build subdirectory location helps.

Then I like docs/build/src/ better. What about docs/build/translated-src/ ?

That current PDF listing page is just a (unattractive) list of files that you have to read line-by-line to find what you want. We can do better than that. We use the same language division and buildup as @grandixximo showed a peek preview of. One style for all doc pages. A better integrated system and more consistent.

Ok :)

It is probably also prudent to separate out the GUIs (like gmoccapy).

You are talking about the documentation of the GUIs?

No, I was talking about the translation of messages in the GUIs' code.

Sure, didn't get what I have read there 🙈

Yeah that would be a good thing to have one file for each GUI and one for general stuff like error messages. So if a user wants only to translate the GUI he/she is using, that makes it much easier.

Three custom Rouge lexers for LinuxCNC's domain-specific languages,
loaded by asciidoctor via -r flags in the build rules.

- rouge_hal.rb: HAL component / pin / signal syntax
- rouge_ngc.rb: G-code (RS-274NGC) with special-case handling for
  LinuxCNC #<_ini[...]NAME> and #<_hal[...]> parameter references
- rouge_ini.rb: INI config syntax with # comment support
- image_resolver.rb: treeprocessor that resolves image targets the way
  the asciidoc-py build did (per-language tree with English fallback).
  Auto-substitutes foo_en.ext -> foo_<lang>.ext.  No DEFAULT_LANGS
  fallback; requires the doc-languages attribute set from po4a.cfg.

- xref_resolver.rb: preprocessor that resolves bare <<anchor,Title>>
  cross-document references to the right per-document HTML target.
- lcnc-overrides.css: theme on top of asciidoctor's default stylesheet.
  Light + dark via prefers-color-scheme (no toggle, no JS).  Heading
  colors #1a3a6c (incl. #header h1 per hansu).  Uniform listing
  background, inline code rounding, dark-mode SVG card wrapper.
- pdf-theme.yml: asciidoctor-pdf theme; CJK + DejaVu fonts.
- docinfo.html: shared docinfo for asciidoctor (docinfo=shared).
- docs/html/index.css, linuxcnc.css, gcode.html: static landing-page
  styling refresh; removes external font fetches.
- docs/src/index.tmpl: master HTML index template polish (JS-free per
  bertho's patch); @TRANSLATIONS@ / @ENDTRANSLATIONS@ gated by build.
Builds a TrueType (glyf-based) copy of the Simplified Chinese cut of
Noto Serif CJK from the system .ttc collection.  prawn 2.4 (used by
asciidoctor-pdf) corrupts the PDF when asked to embed CFF outlines, so
we convert the curves with cu2qu (python3-fonttools) first.

Subsets to the glyph set actually used by the docs to keep the font
small.  Scans both docs/src and the po catalogues, so the subset
holds the translated text too.  Without this, a fresh build's subset
would come up empty and the converter would fall back to the full
~65k-glyph face (~200s vs ~1.5s).
- src/configure.ac: probe ruby-rouge, python3-fonttools, and
  NotoSerifCJK fontconfig path portably.  When BUILD_DOCS_HTML or
  BUILD_DOCS_PDF was requested without --enable-X explicit and a dep
  is missing: warn and disable that target.  When the user explicitly
  passed --enable-build-documentation=pdf with deps missing: fail
  hard (don't silently produce no PDFs).  Same hard-fail for any
  explicit PDF request, not only =pdf.

- debian/configure: regenerate Build-Depends-Indep with asciidoctor
  toolchain (ruby-asciidoctor, ruby-asciidoctor-pdf, ruby-rouge,
  python3-fonttools, fonts-dejavu, fonts-noto-cjk) replacing the
  asciidoc-py / dblatex / xsltproc stack.

- debian/control.top.in: expand @DOC_DEPENDS@ accordingly.
Replace the legacy asciidoc-py + dblatex + xsltproc pipeline with
asciidoctor + asciidoctor-pdf + Rouge.  HTML built via asciidoctor,
PDFs via asciidoctor-pdf, manpages via asciidoctor manpage doctype
(troff) plus html5 backend.  Manual_Pages.pdf assembled from a
generated master document including every PDF_MAN_ORDER entry as a
chapter (leveloffset=+1, hard page break per entry).

- docs/src/Submakefile: rewrite around asciidoctor + Rouge.  Per-
  language target lists derived from po4a.cfg.  Idempotent build:
  stamp-gated copy_asciidoc_files / gen_complist / checkref, content-
  stable mtimes via touch on .pot regen, .SECONDARY for translated
  Master_*.pdf so leaf rules don't churn.  CJK font discovery via
  fontconfig (NotoSerifCJK .ttc) + DejaVu Sans Mono symlinks under
  objects/.fonts/.  Rouge theme rendered from the Rouge API at build
  time, single-mode fallback for older Rouge without light!/dark!.
  Manpage HTML resolves troff .so aliases via symlink, falls back to
  objects/man for halcompile-generated component adocs.

- debian/rules.in: per-lang translated PDF cleanup loop derived from
  po4a.cfg; comment about gcode.html lang tree.

- src/Makefile.inc.in: drop legacy asciidoc-py / dblatex variables.

Drop legacy infrastructure:
  asciidoc-dont-replace-arrows.conf, attribute-colon.conf,
  docbook.conf, docbook-image.conf, emc2.sty, html-images.xslt,
  html-latex-images, image-wildcard, links.xslt, links_db_gen.py,
  xhtml11.conf + xhtml11-{head-foot,latexmath,links}.conf,
  source-highlight/ (replaced by Rouge), scripts/inkscape rsvg shim
  (asciidoctor handles SVG natively).
Adjust .adoc sources where asciidoctor's parser is stricter than the
legacy asciidoc-py, where INI/HAL/NGC examples were malformed enough
that the new Rouge highlighters exposed the breakage, or where blank
lines were missing before * bullet lists (asciidoctor requires them).

Touches docs/src/{code,config,drivers,examples,gcode,gui,hal,ladder,
lathe,motion,plasma,remap,tooldatabase,user}/*.adoc, docs/src/man/man9
manpage sources, docs/help/tklinuxcnc.adoc, and a table cell merge
fix in src/hal/components/lut5.comp.  Also a small per-language
asset swap (control-point_es.svg).
- gen_complist.py: produce content-stable mtime via write_if_changed
  helper and single-write add_links.  Bertho identified that
  components_gen.adoc was being rewritten with HTML-existence-
  dependent content every build, bumping documentation.pot's mtime
  and making po4a look like it ran twice.

- po4a.cfg: master config sourced for per-language target lists in
  the Submakefile (LANGUAGES derives from po4a_langs entry).
  AsciiDoc_def / man_def / xhtml_def / Text_def aliases for the
  asciidoctor pipeline.
- .gitignore: drop nocheck workaround, ignore autom4te.cache,
  .checklink.*.tmp leftovers from docs/src/checkref.
- docs/src/.gitignore: ignore generated objects/.fonts and per-lang
  build leftovers.
- docs/man/images/Images_info.adoc: small note refresh.
Pass --enable-build-documentation-translation to configure so the docs
CI builds the translated HTML/PDF set, not only English.  Exercises
the po4a / per-language pipeline end-to-end on every PR.
@grandixximo grandixximo force-pushed the docs/asciidoctor-migration branch from b8f1ffe to 239dd3f Compare May 30, 2026 02:00
@hansu
Copy link
Copy Markdown
Member

hansu commented May 30, 2026

@grandixximo grandixximo force-pushed the docs/asciidoctor-migration branch from b8f1ffe to 239dd3f 5 hours ago

Thanks!

@BsAtHome
Copy link
Copy Markdown
Contributor

An aesthetic issue that I think should be fixed in a follow-up...
(BTW, you've done a tremendous job and this should not be delayed any further)

In otf2ttf.py:26: The regex content does not render properly if your local font does not support the ideographs in the regex (my system shows boxes in the second range and a "space" in the first). Having i18n properly visible for all is just difficult.

The regex can support Unicode escapes with ur'[\uXXXX-\uYYYY\uZZZZ-\uVVVV]'. It may be a good idea to use that and make sure it always renders in a visible manner. Just add the current UTF-8 encoded regex content in the comment above for reference.

@grandixximo
Copy link
Copy Markdown
Contributor Author

grandixximo commented May 30, 2026

Couldn't have done without you guys, thank you for guiding me.

If this lands next step is this.

grandixximo/linuxcnc@docs/asciidoctor-migration...docs/unified-build-output

I went for adoc instead of translated-src folder, other than that I think the shape is right, we can discuss on that PR whenever this merges, that's next I think

@BsAtHome BsAtHome merged commit c3caef5 into LinuxCNC:master May 30, 2026
15 checks passed
@grandixximo grandixximo deleted the docs/asciidoctor-migration branch May 30, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants