From d46718abd166ede72c38d99630028f75e7ea77c1 Mon Sep 17 00:00:00 2001 From: Leszek Grzanka Date: Tue, 20 Jan 2026 13:43:37 +0100 Subject: [PATCH 1/4] Add simulator management documentation and update navigation --- docs/backend/index.md | 1 + docs/backend/simulator_management.md | 257 +++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 259 insertions(+) create mode 100644 docs/backend/simulator_management.md diff --git a/docs/backend/index.md b/docs/backend/index.md index ae78464..0b3ffdc 100644 --- a/docs/backend/index.md +++ b/docs/backend/index.md @@ -8,6 +8,7 @@ The documenation contains: * [For developers](for_developers.md) - How to build backend for developers * [Using docker](using_docker.md) - How to build backend for deployment using Docker + * [Simulator Management](simulator_management.md) - Managing simulation engine binaries with S3 storage * [API reference](swagger.md) - (useful for frontend development), auto-generated from swagger yaml * [Jobs and tasks](states.md) - Description of states of jobs and task * [Persistent storage](persistency.md) - Description of database model diff --git a/docs/backend/simulator_management.md b/docs/backend/simulator_management.md new file mode 100644 index 0000000..61bf97c --- /dev/null +++ b/docs/backend/simulator_management.md @@ -0,0 +1,257 @@ +# Managing Simulators with S3 Storage + +## Overview + +Yaptide supports two simulation engines: **SHIELD-HIT12A** and **FLUKA**. This document explains the simulator management system, how binaries are stored and deployed, and how to manage different simulator versions. + +## Architecture and Logic + +Yaptide implements a deployment strategy for managing simulation engine binaries. In the demo mode, when S3 environment variables are not configured in the `.env` file (or are missing from the environment), the system automatically downloads a constrained demo version of SHIELD-HIT12A directly from [shieldhit.org](https://shieldhit.org). This demo build is intended for development and validation, uses a fixed random number generator seed, enforces a 10,000 primary particle limit, and remains unencrypted because it is public. + +In production mode, when S3 credentials are configured in the `.env` file, the system retrieves full-featured simulator binaries from S3-compatible object storage. SHIELD-HIT12A and FLUKA binaries are stored in encrypted form; the decryption password and salt are supplied via the `.env` file and must match the values used during upload. This mode provides unrestricted functionality (custom random seeds, unlimited primaries) and retains a graceful fallback: if S3 download fails or credentials are unavailable, the system automatically falls back to the demo SHIELD-HIT12A build. + +## Flow During Container Startup + +The `yaptide-simulation-worker` container is built from [yaptide/Dockerfile-simulation-worker](https://github.com/yaptide/yaptide/blob/master/Dockerfile-simulation-worker) and invokes the startup script [yaptide/run_simulation_worker.sh](https://github.com/yaptide/yaptide/blob/master/run_simulation_worker.sh), which in turn calls the simulator manager [yaptide/yaptide/admin/simulators.py](https://github.com/yaptide/yaptide/blob/master/yaptide/admin/simulators.py). All required configuration is expected in the `.env` file; if a variable is absent there, the runtime environment is consulted, and if still missing a fallback path is taken. + +SHIELD-HIT12A is always initialized: the script first attempts to fetch and decrypt the encrypted binary from S3 using `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_SHIELDHIT_BUCKET`, `S3_SHIELDHIT_KEY`, and the decryption parameters `S3_ENCRYPTION_PASSWORD` and `S3_ENCRYPTION_SALT` from `.env` or the environment. If any of these are unavailable or the download fails, the process falls back to downloading the public demo build from shieldhit.org and installs it into `/simulators/shieldhit12a/bin`. + +FLUKA is initialized only when `S3_FLUKA_BUCKET` and `S3_FLUKA_KEY` are provided in `.env` or the environment; the payload is encrypted and decrypted with the same password and salt variables. If these variables are absent, FLUKA installation is skipped, allowing deployments that rely solely on SHIELD-HIT12A. + +## Simulator Management Command Reference + +!!! warning "Linux/WSL Required" + The simulation binaries are compiled for Linux. All commands should be executed on Linux or within Windows Subsystem for Linux (WSL). Running on native Windows will not work. + +Use the project virtual environment managed by Poetry (see [Backend: For developers](for_developers.md) for installation and activation). The `simulators.py` CLI manages encrypted simulator binaries: it downloads SHIELD-HIT12A (with demo fallback), optionally downloads FLUKA, uploads encrypted artifacts to S3, and provides encrypt/decrypt helpers. Run `--help` to list commands: + +```bash +poetry run ./yaptide/admin/simulators.py --help +``` + +For command-specific options, append `--help`, e.g.: + +```bash +poetry run ./yaptide/admin/simulators.py download-shieldhit --help +``` + + +### Available Commands + +Most S3 parameters are read from environment variables (see [Environment Variables Configuration](#environment-variables-configuration) below). Run each command with `--help` for full options: + +#### Download SHIELD-HIT12A + +Download SHIELD-HIT12A to the `./download` directory from S3 or fall back to demo version: + +```bash +poetry run ./yaptide/admin/simulators.py download-shieldhit --dir ./download --decrypt +``` + +Verify the download: + +```bash +ls -lh ./download +``` + +Check version: +```bash +./download/shieldhit --version +``` + +See options: + +```bash +poetry run ./yaptide/admin/simulators.py download-shieldhit --help +``` + +#### Download FLUKA + +Download FLUKA simulator (encrypted, requires S3 credentials): + +```bash +poetry run ./yaptide/admin/simulators.py download-fluka --dir ./download +``` + +See options: + +```bash +poetry run ./yaptide/admin/simulators.py download-fluka --help +``` + +#### Upload Simulator to S3 + +Upload a compiled binary (optionally encrypted): + +```bash +poetry run ./yaptide/admin/simulators.py upload --bucket my-bucket --file ./shieldhit --encrypt +``` + +See options: + +```bash +poetry run ./yaptide/admin/simulators.py upload --help +``` + +#### Encrypt File + +Manually encrypt a binary: + +```bash +poetry run ./yaptide/admin/simulators.py encrypt --infile ./shieldhit --outfile ./shieldhit.encrypted --password my-pass --salt my-salt +``` + +See options: + +```bash +poetry run ./yaptide/admin/simulators.py encrypt --help +``` + +#### Decrypt File + +Manually decrypt a binary: + +```bash +poetry run ./yaptide/admin/simulators.py decrypt --infile ./shieldhit.encrypted --outfile ./shieldhit --password my-pass --salt my-salt +``` + +See options: + +```bash +poetry run ./yaptide/admin/simulators.py decrypt --help +``` + +## Practical Examples + +### Uploading a New SHIELD-HIT12A Version + +Assume you have compiled a new SHIELD-HIT12A binary and want to upload it to S3 for production use. + +**Step 1: Compile the binary** + +```bash +./compile_shieldhit.sh +# Result: ./build/shieldhit (or shieldhit.exe on Windows) +``` + +**Step 2: Encrypt the binary** + +```bash +poetry run ./yaptide/admin/simulators.py encrypt \ + --infile ./build/shieldhit \ + --outfile ./build/shieldhit.encrypted \ + --password my-secure-password \ + --salt my-salt-value +``` + +**Step 3: Upload encrypted binary to S3** + +```bash +poetry run ./yaptide/admin/simulators.py upload \ + --bucket yaptide-simulators \ + --file ./build/shieldhit.encrypted \ + --endpoint s3.mycompany.com \ + --access-key AKIAIOSFODNN7EXAMPLE \ + --secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +``` + +### Uploading a New FLUKA Version + +**Step 1: Compile FLUKA and create archive** + +```bash +tar -czf fluka_v2024.tar.gz fluka/ +``` + +**Step 2: Encrypt the archive** + +```bash +poetry run ./yaptide/admin/simulators.py encrypt \ + --infile fluka_v2024.tar.gz \ + --outfile fluka_v2024.tar.gz.encrypted \ + --password my-secure-password \ + --salt my-salt-value +``` + +**Step 3: Upload to S3** + +```bash +poetry run ./yaptide/admin/simulators.py upload \ + --bucket yaptide-simulators-fluka \ + --file fluka_v2024.tar.gz.encrypted \ + --endpoint s3.mycompany.com \ + --access-key AKIAIOSFODNN7EXAMPLE \ + --secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +``` + +## Environment Variables Configuration + +For S3-based deployment, configure these environment variables (`.env` file): + +```bash +# S3 General Configuration +S3_ENDPOINT=s3.mycompany.com +S3_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE +S3_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + +# SHIELD-HIT12A Configuration +S3_SHIELDHIT_BUCKET=yaptide-simulators +S3_SHIELDHIT_KEY=shieldhit_latest + +# FLUKA Configuration +S3_FLUKA_BUCKET=yaptide-simulators-fluka +S3_FLUKA_KEY=fluka_v2024.tar.gz.encrypted + +# Encryption Configuration +S3_ENCRYPTION_PASSWORD=my-secure-password +S3_ENCRYPTION_SALT=my-salt-value +``` + +!!! note "Empty .env in Development" + During local development, you can leave these variables unset. The system will automatically use demo SHIELD-HIT12A version from shieldhit.org. + +## Troubleshooting + +### Demo Version Download Fails + +If you see "SHIELD-HIT12A download failed", the demo version download from shieldhit.org failed: + +1. Check internet connectivity +2. Verify that [shieldhit.org](https://shieldhit.org) is accessible +3. The demo version URL might have changed - check the code in `simulator_storage.py` + +### S3 Connection Error + +``` +No credentials found. Check your access key and secret key. +``` + +**Solution**: Verify `.env` file contains correct S3 credentials. + +### Decryption Failed + +``` +Decryption failed - invalid token (password+salt) +``` + +**Solutions**: +- Verify password and salt are correct +- Ensure the file was encrypted with the same password/salt +- Ensure the file is not corrupted + +### S3 Bucket Not Found + +``` +Problem accessing bucket named: my-bucket +``` + +**Solutions**: +- Verify bucket exists on S3 +- Verify S3_ENDPOINT is correct +- Verify credentials have permissions to access the bucket + +## See Also + +- [Using Docker](using_docker.md) - Deployment instructions +- [For Developers](for_developers.md) - Local development setup +- YAPTIDE GitHub: [yaptide/yaptide](https://github.com/yaptide/yaptide) diff --git a/mkdocs.yml b/mkdocs.yml index b2d9e39..f0e7341 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,6 +19,7 @@ nav: - Overview: backend/index.md - For developers: backend/for_developers.md - Using docker: backend/using_docker.md + - Simulator Management: backend/simulator_management.md - API: backend/swagger.md - Jobs and tasks: backend/states.md - Persistent storage: backend/persistency.md From 314bc55971e58a38dd2e03047f57f12ce0a1ee2f Mon Sep 17 00:00:00 2001 From: Leszek Grzanka Date: Tue, 20 Jan 2026 14:46:32 +0100 Subject: [PATCH 2/4] Update simulator management documentation and quick start instructions --- docs/backend/simulator_management.md | 164 +++++++++++---------------- docs/documentation/index.md | 30 +++-- 2 files changed, 89 insertions(+), 105 deletions(-) diff --git a/docs/backend/simulator_management.md b/docs/backend/simulator_management.md index 61bf97c..8b74a48 100644 --- a/docs/backend/simulator_management.md +++ b/docs/backend/simulator_management.md @@ -20,8 +20,7 @@ FLUKA is initialized only when `S3_FLUKA_BUCKET` and `S3_FLUKA_KEY` are provided ## Simulator Management Command Reference -!!! warning "Linux/WSL Required" - The simulation binaries are compiled for Linux. All commands should be executed on Linux or within Windows Subsystem for Linux (WSL). Running on native Windows will not work. +**Note:** The simulation binaries are compiled for Linux. All commands should be executed on Linux or within Windows Subsystem for Linux (WSL). Running on native Windows will not work. Use the project virtual environment managed by Poetry (see [Backend: For developers](for_developers.md) for installation and activation). The `simulators.py` CLI manages encrypted simulator binaries: it downloads SHIELD-HIT12A (with demo fallback), optionally downloads FLUKA, uploads encrypted artifacts to S3, and provides encrypt/decrypt helpers. Run `--help` to list commands: @@ -40,54 +39,62 @@ poetry run ./yaptide/admin/simulators.py download-shieldhit --help Most S3 parameters are read from environment variables (see [Environment Variables Configuration](#environment-variables-configuration) below). Run each command with `--help` for full options: -#### Download SHIELD-HIT12A +#### Download SHIELD-HIT12A from S3 (with demo fallback) -Download SHIELD-HIT12A to the `./download` directory from S3 or fall back to demo version: +Retrieve the SHIELD-HIT12A binary from S3 storage and decrypt it locally. If S3 is unavailable, the system falls back to the public demo version: ```bash poetry run ./yaptide/admin/simulators.py download-shieldhit --dir ./download --decrypt ``` -Verify the download: +Confirm successful download by listing the directory: ```bash ls -lh ./download ``` -Check version: +Verify the binary version matches your expectations: ```bash ./download/shieldhit --version ``` -See options: +Optionally validate the binary inside the simulation worker container (mounting read-only at a non-standard path to isolate from production simulators): +```bash +docker run --rm -it \ + -v "$(pwd)/download/shieldhit:/opt/test/shieldhit:ro" \ + --entrypoint /bin/bash yaptide_simulation_worker \ + -c "/opt/test/shieldhit --version" +``` + +For additional configuration options and parameters: ```bash poetry run ./yaptide/admin/simulators.py download-shieldhit --help ``` -#### Download FLUKA +#### Retrieve FLUKA from S3 (encrypted) -Download FLUKA simulator (encrypted, requires S3 credentials): +Download and decrypt the FLUKA binary from S3 storage. Requires valid S3 credentials configured in `.env`: ```bash poetry run ./yaptide/admin/simulators.py download-fluka --dir ./download ``` -See options: +For additional configuration options: ```bash poetry run ./yaptide/admin/simulators.py download-fluka --help ``` -#### Upload Simulator to S3 +#### Upload Simulator Binary to S3 -Upload a compiled binary (optionally encrypted): +Upload a compiled simulator binary to S3-compatible storage with optional encryption: ```bash poetry run ./yaptide/admin/simulators.py upload --bucket my-bucket --file ./shieldhit --encrypt ``` -See options: +For additional configuration options: ```bash poetry run ./yaptide/admin/simulators.py upload --help @@ -125,63 +132,77 @@ poetry run ./yaptide/admin/simulators.py decrypt --help ### Uploading a New SHIELD-HIT12A Version -Assume you have compiled a new SHIELD-HIT12A binary and want to upload it to S3 for production use. +Step-by-step example assuming SHIELD-HIT12A sources live in `$HOME/workspace/shieldhit`. -**Step 1: Compile the binary** +**1) Compile the binary (from the source dir)** +Enter the source tree: ```bash -./compile_shieldhit.sh -# Result: ./build/shieldhit (or shieldhit.exe on Windows) +cd "$HOME/workspace/shieldhit" ``` -**Step 2: Encrypt the binary** +Compile with gfortran: +```bash +make gfortran -j +``` + +After build, the binary should be `./shieldhit`. Check its version: +Verify binary version: ```bash -poetry run ./yaptide/admin/simulators.py encrypt \ - --infile ./build/shieldhit \ - --outfile ./build/shieldhit.encrypted \ - --password my-secure-password \ - --salt my-salt-value +./shieldhit --version +# Expected shape (example): +# SHIELD-HIT12A +# Version: v1.1.0-8-g4ea3f147 +# Build date: Tue, 20 Jan 2026 11:28:34 +0100 +# SHIELD-HIT12A is up to date. ``` -**Step 3: Upload encrypted binary to S3** +Optionally rename to capture host/build metadata (recommended): ```bash -poetry run ./yaptide/admin/simulators.py upload \ - --bucket yaptide-simulators \ - --file ./build/shieldhit.encrypted \ - --endpoint s3.mycompany.com \ - --access-key AKIAIOSFODNN7EXAMPLE \ - --secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +mv ./shieldhit ./shieldhit-lenovo-dev-g12fd3b8c-make-gfortran +``` + +**2) Upload to S3 with encryption (run from project root `yaptide/`)** + +Switch to project root: +```bash +cd "$HOME/workspace/yaptide" ``` -### Uploading a New FLUKA Version +Upload with encryption to S3: +```bash +poetry run ./yaptide/admin/simulators.py upload \ + --bucket shieldhit \ + --file "$HOME/workspace/shieldhit/shieldhit-lenovo-dev-g12fd3b8c-make-gfortran" \ + --encrypt +``` -**Step 1: Compile FLUKA and create archive** +**3) Verify download and execution** +Prepare temp download directory: ```bash -tar -czf fluka_v2024.tar.gz fluka/ +mkdir -p /tmp/sh-download ``` -**Step 2: Encrypt the archive** +Download and decrypt from S3: +```bash +poetry run ./yaptide/admin/simulators.py download-shieldhit --dir /tmp/sh-download --decrypt +``` +Run version check on downloaded binary: ```bash -poetry run ./yaptide/admin/simulators.py encrypt \ - --infile fluka_v2024.tar.gz \ - --outfile fluka_v2024.tar.gz.encrypted \ - --password my-secure-password \ - --salt my-salt-value +/tmp/sh-download/shieldhit --version ``` -**Step 3: Upload to S3** +If you want to validate inside the simulation worker container (without overwriting in-container simulators): ```bash -poetry run ./yaptide/admin/simulators.py upload \ - --bucket yaptide-simulators-fluka \ - --file fluka_v2024.tar.gz.encrypted \ - --endpoint s3.mycompany.com \ - --access-key AKIAIOSFODNN7EXAMPLE \ - --secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +docker run --rm -it \ + -v "/tmp/sh-download/shieldhit:/opt/test/shieldhit:ro" \ + --entrypoint /bin/bash yaptide_simulation_worker \ + -c "/opt/test/shieldhit --version" ``` ## Environment Variables Configuration @@ -205,53 +226,4 @@ S3_FLUKA_KEY=fluka_v2024.tar.gz.encrypted # Encryption Configuration S3_ENCRYPTION_PASSWORD=my-secure-password S3_ENCRYPTION_SALT=my-salt-value -``` - -!!! note "Empty .env in Development" - During local development, you can leave these variables unset. The system will automatically use demo SHIELD-HIT12A version from shieldhit.org. - -## Troubleshooting - -### Demo Version Download Fails - -If you see "SHIELD-HIT12A download failed", the demo version download from shieldhit.org failed: - -1. Check internet connectivity -2. Verify that [shieldhit.org](https://shieldhit.org) is accessible -3. The demo version URL might have changed - check the code in `simulator_storage.py` - -### S3 Connection Error - -``` -No credentials found. Check your access key and secret key. -``` - -**Solution**: Verify `.env` file contains correct S3 credentials. - -### Decryption Failed - -``` -Decryption failed - invalid token (password+salt) -``` - -**Solutions**: -- Verify password and salt are correct -- Ensure the file was encrypted with the same password/salt -- Ensure the file is not corrupted - -### S3 Bucket Not Found - -``` -Problem accessing bucket named: my-bucket -``` - -**Solutions**: -- Verify bucket exists on S3 -- Verify S3_ENDPOINT is correct -- Verify credentials have permissions to access the bucket - -## See Also - -- [Using Docker](using_docker.md) - Deployment instructions -- [For Developers](for_developers.md) - Local development setup -- YAPTIDE GitHub: [yaptide/yaptide](https://github.com/yaptide/yaptide) +``` \ No newline at end of file diff --git a/docs/documentation/index.md b/docs/documentation/index.md index 3060ce7..7ff5380 100644 --- a/docs/documentation/index.md +++ b/docs/documentation/index.md @@ -44,29 +44,41 @@ The deployment includes generation of test coverage report and API reference doc ### Prerequisites First, user needs to install [poetry](https://python-poetry.org). -Then, user needs to install the dependencies for the backend and the documentation: + +### Quick Start Recipe + +**Step 1: Clone and navigate to the repository** ```bash -poetry install --only main,docs +git clone https://github.com/yaptide/for_developers.git +cd for_developers ``` -### Building the documentation +**Step 2: Install dependencies** -To build the documentation run the following command: +```bash +poetry install +``` + +**Step 3: Serve documentation locally** ```bash -poetry run mkdocs build +poetry run mkdocs serve ``` -this will generate the documentation in the `site` folder. +This will start a local webserver on port 8000. Open http://localhost:8000 in your browser to view the documentation. + +The documentation will automatically reload when you modify markdown files. + +### Building the documentation -To serve the documentation locally run the following command: +To build static HTML documentation run: ```bash -poetry run mkdocs serve +poetry run mkdocs build ``` -This will start a local webserver on port 8000. The documentation can be viewed by opening the following url in a browser: http://localhost:8000 +This will generate the documentation in the `site` folder. ### Working with the technical documentation From f000e2cc12adf7e1e1ea6326af711aa0920ad769 Mon Sep 17 00:00:00 2001 From: Leszek Grzanka Date: Tue, 20 Jan 2026 14:47:38 +0100 Subject: [PATCH 3/4] Update quick start instructions for cloning the repository --- docs/documentation/index.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/documentation/index.md b/docs/documentation/index.md index 7ff5380..e52817f 100644 --- a/docs/documentation/index.md +++ b/docs/documentation/index.md @@ -47,10 +47,15 @@ First, user needs to install [poetry](https://python-poetry.org). ### Quick Start Recipe -**Step 1: Clone and navigate to the repository** +**Step 1: Clone the repository** ```bash git clone https://github.com/yaptide/for_developers.git +``` + +Navigate to the repository: + +```bash cd for_developers ``` From c72e042ad237885a8c77b3b21572fd4a409ccb09 Mon Sep 17 00:00:00 2001 From: Leszek Grzanka Date: Tue, 20 Jan 2026 14:52:53 +0100 Subject: [PATCH 4/4] Update simulator binary names in S3 upload instructions --- docs/backend/simulator_management.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/backend/simulator_management.md b/docs/backend/simulator_management.md index 8b74a48..5f4aa03 100644 --- a/docs/backend/simulator_management.md +++ b/docs/backend/simulator_management.md @@ -161,7 +161,7 @@ Verify binary version: Optionally rename to capture host/build metadata (recommended): ```bash -mv ./shieldhit ./shieldhit-lenovo-dev-g12fd3b8c-make-gfortran +mv ./shieldhit ./shieldhit-lenovo-dev-g4ea3f147-make-gfortran ``` **2) Upload to S3 with encryption (run from project root `yaptide/`)** @@ -175,7 +175,7 @@ Upload with encryption to S3: ```bash poetry run ./yaptide/admin/simulators.py upload \ --bucket shieldhit \ - --file "$HOME/workspace/shieldhit/shieldhit-lenovo-dev-g12fd3b8c-make-gfortran" \ + --file "$HOME/workspace/shieldhit/shieldhit-lenovo-dev-g4ea3f147-make-gfortran" \ --encrypt ```