Big rewamp of Documentation with many new additions

.github/workflows/httomo_docs.yml

@@ -8,6 +8,10 @@ on:
   push:
     branches:
       - main
+    tags:
+      - 'v*'
+  schedule:
+    - cron: '0 6 * * 1'
 
 jobs:
   build-docs-publish:
@@ -33,8 +37,7 @@ jobs:
             init-shell: bash
 
         - name: Install httomo-backends
-          run: |
-            pip install --no-deps httomo-backends
+          run: pip install --no-deps -r ./docs/source/doc-pip-requirements.txt
 
         - name: Generate full yaml pipelines using pipeline directives
           run: |
@@ -47,10 +50,18 @@ jobs:
             zip -r pipelines_full_artifact.zip ./docs/source/pipelines_full/
 
         - name: Build docs
-          run: sphinx-build -a -E -b html ./docs/source/ ./docs/build/
+          run: sphinx-build -W --keep-going -a -E -b html ./docs/source/ ./docs/build/
+
+        - name: Check external links
+          if: github.event_name == 'schedule'
+          run: sphinx-build -W --keep-going -a -E -b linkcheck ./docs/source/ ./docs/linkcheck/
 
         - name: Publish docs
-          if: github.ref_type == 'tag' || github.ref_name == 'main'
+          if: >-
+            (github.event_name == 'push' &&
+            (github.ref_type == 'tag' || github.ref_name == 'main')) ||
+            (github.event_name == 'workflow_dispatch' &&
+            github.ref_name == 'main')
           run: ghp-import -n -p -f ./docs/build
           env:
             GITHUB_TOKEN: ${{ github.token }}
@@ -59,4 +70,4 @@ jobs:
           uses: actions/upload-artifact@v4
           with:
             name: full_pipelines-artifact
-            path: pipelines_full_artifact.zip
+            path: pipelines_full_artifact.zip

README.rst

@@ -1,26 +1,25 @@
-HTTomo (High Throughput Tomography pipeline)
-*******************************************************
+High Throughput Tomography software
+***********************************
 
-HTTomo is a user interface (UI) written in Python for fast big data processing using MPI protocols.
-It orchestrates I/O data operations and enables processing on a CPU and/or a GPU. HTTomo utilises other libraries, such as `TomoPy <https://tomopy.readthedocs.io>`_ and `HTTomolibgpu <https://github.com/DiamondLightSource/httomolibgpu>`_
-as backends for data processing. The methods from the libraries are exposed through YAML templates to enable fast task programming.
-
-Installation
-============
-See detailed instructions for `installation <https://diamondlightsource.github.io/httomo/howto/installation.html>`_ .
+HTTomo is a Python framework for high-performance tomographic data processing mainly targeting GPU-compute.
+It orchestrates distributed I/O and CPU/GPU workflows using MPI, while providing
+YAML-based access to processing methods from libraries such as
+`TomoPy <https://tomopy.readthedocs.io>`_ and `HTTomolibgpu <https://github.com/DiamondLightSource/httomolibgpu>`_.
 
 Documentation
-==============
-Please check the full `documentation <https://diamondlightsource.github.io/httomo/>`_.
+=============
+
+The `HTTomo documentation <https://diamondlightsource.github.io/httomo/>`_
+contains installation instructions, a quickstart, ready-to-use pipelines,
+reference material and developer guidance.
+
+After installation, use :code:`python -m httomo --help` to inspect the command
+line interface. A typical workflow is to validate a pipeline and then run it:
 
-Running HTTomo:
-================
+.. code-block:: console
 
-* Install the module following any chosen `installation <https://diamondlightsource.github.io/httomo/howto/installation.html>`_ path.
-* For help with the command line interface, execute :code:`python -m httomo --help`
-* Choose the existing `YAML pipeline <https://diamondlightsource.github.io/httomo/pipelines/yaml.html>`_ or build a new one using ready-to-be-used `templates <https://diamondlightsource.github.io/httomo/backends/templates.html>`_.
-* Optional: perform the validity check of the YAML pipeline file with the `YAML checker <https://diamondlightsource.github.io/httomo/utilities/yaml_checker.html>`_.
-* Run HTTomo with :code:`python -m httomo run [OPTIONS] IN_DATA_FILE YAML_CONFIG OUT_DIR`, see more on that `here <https://diamondlightsource.github.io/httomo/howto/run_httomo.html>`_.
+   python -m httomo check pipeline.yaml input.nxs
+   python -m httomo run input.nxs pipeline.yaml output_directory
 
 Release Tagging Scheme
 ======================

docs/build_sphinx_docs.bat

@@ -0,0 +1,44 @@
+@echo off
+setlocal
+
+echo ********************************************************************************
+echo                         Starting the Sphinx script
+echo ********************************************************************************
+echo                 Creating plugin API files and HTML pages...
+
+rem Resolve all paths relative to the directory containing this batch file.
+set "SCRIPT_DIR=%~dp0"
+
+rem Remove old generated API files and build output to avoid obsolete files.
+if exist "%SCRIPT_DIR%source\developers\generated" (
+    rmdir /s /q "%SCRIPT_DIR%source\developers\generated"
+    if errorlevel 1 goto :cleanup_failed
+)
+
+if exist "%SCRIPT_DIR%build" (
+    rmdir /s /q "%SCRIPT_DIR%build"
+    if errorlevel 1 goto :cleanup_failed
+)
+
+rem -a writes all output files.
+rem -E rebuilds the Sphinx environment without using the saved cache.
+rem -b html selects the HTML builder.
+rem -W treats warnings as errors; --keep-going reports all warnings in one run.
+sphinx-build -W --keep-going -a -E -b html "%SCRIPT_DIR%source" "%SCRIPT_DIR%build"
+set "EXIT_CODE=%ERRORLEVEL%"
+
+if not "%EXIT_CODE%"=="0" (
+    echo.
+    echo ERROR: Sphinx build failed with exit code %EXIT_CODE%.
+) else (
+    echo.
+    echo Sphinx documentation built successfully.
+    echo Output: "%SCRIPT_DIR%build\index.html"
+)
+
+exit /b %EXIT_CODE%
+
+:cleanup_failed
+echo.
+echo ERROR: Unable to remove an existing generated or build directory.
+exit /b 1