Skip to content

Harness improvements #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andreea-alexandru wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Harness improvements #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5bd00ca
clean arguments, enable num_runs, supress warnings, add mini-workload…
andreea-alexandru Jul 6, 2026
be5fbbe
updated harness output, added measurements,updated README
andreea-alexandru Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
*/build/
io/
third_party/
measurements/
harness/__pycache__/
submission/target/

Expand Down
136 changes: 72 additions & 64 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,65 +1,63 @@
# FHE Benchmarking template
# AES Transciphering FHE Benchmark

## Cloning the workload

```console
git clone https://github.com/fhe-benchmarking/sample-workload
cd sample-workload
git clone https://github.com/code-perspective/temp-fhe-transciphering
cd temp-fhe-transciphering
```

## Dependencies

The harness requires python and some corresponding packages specified in `requirements.txt`.
The harness is written in Python and requires a few packages listed in `requirements.txt`.
```console
python3 -m venv virtualenv
source ./virtualenv/bin/activate
pip3 install -r requirements.txt
```

In this template, the library for the encrypted computations is OpenFHE v1.3.1.
If other libraries are required, the developer should include the relevant instructions for installing.
In this template, the submission (the encrypted computation) is written in Rust and uses
the [tfhe-rs](https://github.com/zama-ai/tfhe-rs) library (v0.5.3) together with the
local `auto-base-conv` crate under `submission/cbs_lib`.

This template assumes the correct version of OpenFHE is installed locally at `/third-party/openfhe`.

### Installing OpenFHE

The installation steps for OpenFHE are described [here](https://openfhe-development.readthedocs.io/en/latest/sphinx_rsts/intro/installation/installation.html).

If OpenFHE
is installed at a different location, that location should be specified using the `-CMAKE_PREFIX_PATH` variable in [`build_task.sh`](https://github.com/fhe-benchmarking/sample-workload/blob/baa3654cdc4a78ce331f49bda1ff48465a9c575d/scripts/build_task.sh#L24).
(In the case of a system-wide installation at `/usr/local/`, unset the `-CMAKE_PREFIX_PATH` variable.)

For users who want to do a local fresh install, they should run `get_openfhe.sh`, which
is designed to install the specified version of OpenFHE at the `third-party` subdirectory in the current directory.
By default, `build_task.sh` looks for the library at this location. See more instructions in `submission/CMakeLists.txt` if
`build_task.sh` does not succeed.

```console
./scripts/get_openfhe.sh
```
Building requires a Rust toolchain (`cargo`/`rustc`). You do not need to install it manually:
[`scripts/build_task.sh`](scripts/build_task.sh) installs the Rust toolchain via `rustup`
if it is not already present, and then builds the submission with `cargo build --release`.
The harness invokes this script automatically, so the first run may take a while as the
dependencies are compiled.

## Running the "AES transciphering" workload

The goal of this workload is to homomorphically transcipher a number of AES blocks into FHE ciphertexts encrypting the same plaintexts. We assess the "quality" of the resulting FHE ciphertexts by evaluating two mini-workloads after the transciphering: 1. Computing the maximum between the input message parsed as 16-bit unsigned integers; and 2. Computing the inner product modulo 2^16 of the first half of the input message parsed as 16-bit unsigned integers and the second half.
The goal of this workload is to homomorphically transcipher a number of AES blocks into FHE
ciphertexts encrypting the same plaintexts. We assess the "quality" of the resulting FHE
ciphertexts by evaluating two mini-workloads after the transciphering: 1. Computing the
maximum between the input message parsed as 16-bit unsigned integers; and 2. Computing the
inner product modulo 2^16 of the first half of the input message parsed as 16-bit unsigned
integers and the second half.

An example run is provided below.
The mini-workload is selected with `--mini_workload` (`0` for the maximum, `1`
for the inner product). An example run is provided below.

```console
$ python3 harness/run_submission.py -h
usage: run_submission.py [-h] [--num_runs NUM_RUNS] [--seed SEED] [--clrtxt CLRTXT] {0,1,2,3}
usage: run_submission.py [-h] [--num_runs NUM_RUNS] [--seed SEED]
[--mini_workload MINI_WORKLOAD]
{0,1,2,3}

Run the AES transciphering FHE benchmark.
Run the AES-transciphering FHE benchmark.

positional arguments:
{0,1,2,3} Instance size (0-toy/1-small/2-medium)
{0,1,2,3} Instance size (0-toy/1-small/2-medium/3-large)

options:
-h, --help show this help message and exit
--num_runs NUM_RUNS Number of times to run steps 4-9 (default: 1)
--seed SEED Random seed for dataset and query generation
--clrtxt CLRTXT Specify with 1 if to rerun the cleartext computation
-h, --help show this help message and exit
--num_runs NUM_RUNS Number of times to run steps 4-9 (default: 1)
--seed SEED Random seed for dataset generation
--mini_workload MINI_WORKLOAD
Specify 0 for mini workload = max and 1 for mini
workload = dot product.

$ python3 ./harness/run_submission.py 2 --seed 3 --num_runs 2
$ python3 ./harness/run_submission.py 1 --seed 3 --num_runs 2 --mini_workload 0

[...]

Expand All @@ -76,59 +74,69 @@ deactivate
```bash
Each submission to the workload in the FHE benchmarking should have the following directory structure:
[root] /
| ├─datasets/ # Holds cleartext data
| | ├─ toy/ # each instance-size in in a separate subdirectory
| ├─datasets/ # Holds cleartext data (plaintext db, AES key/IV, AES-encrypted db, expected outputs)
| | ├─ toy/ # each instance-size in a separate subdirectory
| | ├─ small/
| | ├─ medium/
| | ├─ large/
| ├─docs/ # Documentation (beyond the top-level README.md)
| ├─harness/ # Scripts to generate data, run workload, check results
| ├─build/ # Handle installing dependencies and building the project
| ├─scripts/ # Handle installing dependencies and building the project (build_task.sh)
| ├─submission/ # The implementation, this is what the submitters modify
| | └─ README.md # likely also a src/, include/ subdirectories, CMakeLists.txt, etc.
| | ├─ cbs_lib/ # The `auto-base-conv` Rust crate used by the submission
| | ├─ src/ # Rust sources for the client_*/server_* executables
| | └─ Cargo.toml # Rust package manifest
| ├─io/ # Directory to hold the I/O between client & server parts
| | ├─ toy/ # The reference implementation has subdirectories
| | ├─ public_keys/ # holds the public evaluation keys
| | ├─ ciphertexts_download/ # holds the ciphertexts to be downloaded by the client
| | ├─ ciphertexts_upload/ # holds the ciphertexts (or other data except keys) to be uploaded by the client
| | ├─ intermediate/ # internal information to be passed around the functions
| | └─ secret_key/ # holds the secret key
| | ├─ secret_keys/ # holds the secret key(s)
| | ├─ ciphertexts_upload/ # holds the ciphertexts (or other data except keys) to be uploaded by the client
| | ├─ ciphertexts_download/ # holds the mini-workload ciphertexts to be downloaded by the client
| | ├─ ciphertext_aes_download/ # holds the transciphered (AES-decrypted) ciphertexts to be downloaded by the client
| | └─ intermediate/ # internal information to be passed around the functions
| | ├─ small/
| | …
| | ├─ medium/
| | …
| ├─measurements/ # Holds json files with the results for each run
| | ├─ toy/ # each instance-size in in a separate subdirectory
| ├─measurements/ # Holds json files (results-<run>.json) with the results for each run
| | ├─ toy/ # each instance-size in a separate subdirectory
| | ├─ small/
| | ├─ medium/
```

## Description of stages

A submitter can edit any of the `client_*` / `server_*` sources in `/submission`.
A submitter can edit any of the `client_*` / `server_*` sources in `/submission`.
Moreover, for the particular parameters related to a workload, the submitter can modify the params files.
If the current description of the files are inaccurate, the stage names in `run_submission` can be also
If the current description of the files are inaccurate, the stage names in `run_submission` can be also
modified.

The current stages are the following, targeted to a client-server scenario.
The order in which they are happening in `run_submission` assumes an initialization step which is
database-dependent and run only once, and potentially multiple runs for multiple queries.
Each file can take as argument the test case size.


| Stage executables | Description |
|----------------------------------|-------------|
| `client_key_generation` | Generate all key material and cryptographic context at the client.
| `client_preprocess_dataset` | (Optional) Any in the clear computations the client wants to apply over the dataset/model.
| `client_preprocess_query` | (Optional) Any in the clear computations the client wants to apply over the query/input.
| `client_encode_encrypt_db` | (Optional) Plaintext encoding and encryption of the dataset/model at the client.
| `client_encode_encrypt_query` | Plaintext encoding and encryption of the query/input at the client.
| `server_preprocess_dataset` | (Optional) Any in the clear or encrypted computations the server wants to apply over the dataset/model.
| `server_encrypted_compute` | The computation the server applies to achieve the workload solution over encrypted daa.
| `client_decrypt_decode` | Decryption and plaintext decoding of the result at the client.
| `client_postprocess`: | Any in the clear computation that the client wants to apply on the decrypted result.

The order in which they happen in `run_submission` assumes an initialization phase which is
dataset-dependent and run only once, followed by the encrypted computation, which can be repeated
`--num_runs` times to average out run-to-run variability in the measured latency. (The computation is
deterministic, so the results are identical across runs; only the timings vary.)
Each executable takes the test-case size as its argument.


| Stage executables | Description |
|---------------------------------------|-------------|
| `client_preprocess` | Any in-the-clear computation the client wants to apply over the dataset before encryption.
| `client_key_generation` | Generate all key material and cryptographic context at the client.
| `client_encode_encrypt` | Plaintext encoding and encryption of the AES key at the client.
| `server_preprocess_dataset` | (Optional) Any in-the-clear or encrypted computation the server applies over the AES-encrypted dataset.
| `server_encrypted_aes_decryption` | Homomorphically decrypt the AES ciphertexts, producing FHE ciphertexts of the same plaintexts (the transciphering).
| `server_encrypted_compute` | Evaluate the selected mini-workload (maximum or inner product) over the transciphered ciphertexts.
| `client_decrypt_decode_aes_decryption`| Decryption and plaintext decoding of the transciphered result at the client.
| `client_postprocess_aes_decryption` | Any in-the-clear computation over the decrypted transciphering result (written to `result_aes.txt`).
| `client_decrypt_decode` | Decryption and plaintext decoding of the mini-workload result at the client.
| `client_postprocess` | Any in-the-clear computation over the decrypted mini-workload result (written to `result.txt`).

The harness additionally runs the Python helpers `generate_dataset.py` (generate and AES-encrypt the
dataset), `cleartext_impl.py` (cleartext reference), and `verify_aes_decryption.py` / `verify_result.py`
(correctness oracles).

The outer python script measures the runtime of each stage.
The current stage separation structure requires reading and writing to files more times than minimally necessary.
For a more granular runtime measuring, which would account for the extra overhead described above, we encourage
submitters to separate and print in a log the individual times for reads/writes and computations inside each stage.
submitters to separate and print in a log the individual times for reads/writes and computations inside each stage.
2 changes: 1 addition & 1 deletion harness/cleartext_impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

def main():

__, params, __, __, __, __ = parse_submission_arguments('Generate dataset for FHE benchmark.')
__, params, __, __, __ = parse_submission_arguments('Generate dataset for FHE benchmark.')
DATASET_PATH = params.datadir() / f"db.txt"
DATASET_ENC_PATH = params.datadir() / f"db.hex"
AES_KEY_PATH = params.datadir() / f"aes_key.hex"
Expand Down
2 changes: 1 addition & 1 deletion harness/generate_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from utils import parse_submission_arguments

def main():
__, params, seed, __, __, __ = parse_submission_arguments('Generate dataset for FHE benchmark.')
__, params, seed, __, __ = parse_submission_arguments('Generate dataset for FHE benchmark.')
DATASET_PATH = params.datadir() / f"db.txt"
AES_KEY_PATH = params.datadir() / f"aes_key.hex"
IV_PATH = params.datadir() / f"aes_iv.hex"
Expand Down
119 changes: 64 additions & 55 deletions harness/run_submission.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def main():

# 0. Prepare running
# Get the arguments
size, params, seed, num_runs, clrtxt, mini_workload = utils.parse_submission_arguments('Run the add-two-values FHE benchmark.')
size, params, seed, num_runs, mini_workload = utils.parse_submission_arguments('Run the AES-transciphering FHE benchmark.')
test = instance_name(size)
print(f"\n[harness] Running submission for {test} dataset")

Expand Down Expand Up @@ -83,60 +83,69 @@ def main():
subprocess.run([exec_dir/"server_preprocess_dataset", str(size)], check=True)
utils.log_step(6, "(Encrypted) dataset preprocessing")

# 7. Server side: Run aes_decryption
subprocess.run([exec_dir/"server_encrypted_aes_decryption", str(size)], check=True)
utils.log_step(7, "Encrypted aes decryption")
utils.log_size(io_dir / "ciphertext_aes_download", "Encrypted results")

# 8. Server side: Run the encrypted processing run exec_dir/server_encrypted_compute
subprocess.run([exec_dir/"server_encrypted_compute", str(size)], check=True)
utils.log_step(8, "Encrypted computation of mini workload")
utils.log_size(io_dir / "ciphertexts_download", "Encrypted results")

# 9. Client-side: decrypt
subprocess.run([exec_dir/"client_decrypt_decode_aes_decryption", str(size)], check=True)
utils.log_step(9, "Result decryption")

# 10. Client-side: post-process
subprocess.run([exec_dir/"client_postprocess_aes_decryption", str(size)], check=True)
utils.log_step(10, "Result postprocessing")

# 11. Client-side: decrypt
subprocess.run([exec_dir/"client_decrypt_decode", str(size)], check=True)
utils.log_step(11, "Result decryption")

# 12. Client-side: post-process
subprocess.run([exec_dir/"client_postprocess", str(size)], check=True)
utils.log_step(12, "Result postprocessing")

# 13. Verify aes_decryption result
aes_expected_file = params.datadir() / "expected_aes.txt"
aes_result_file = io_dir / "result_aes.txt"

if not aes_result_file.exists():
print(f"Error: Result file {aes_result_file} not found")
sys.exit(1)

subprocess.run(["python3", harness_dir/"verify_aes_decryption.py",
str(aes_expected_file), str(aes_result_file)], check=False)

# 14. Verify the final result
expected_file = params.datadir() / "max_value.txt"
if mini_workload == 1:
expected_file = params.datadir() / "inner_product.txt"
result_file = io_dir / "result.txt"

if not result_file.exists():
print(f"Error: Result file {result_file} not found")
sys.exit(1)

subprocess.run(["python3", harness_dir/"verify_result.py",
str(expected_file), str(result_file)], check=False)

# 15. Store measurements
run_path = params.measuredir() / f"results.json"
run_path.parent.mkdir(parents=True, exist_ok=True)
utils.save_run(run_path)
# Run steps 7-14 multiple times if requested. The computation is deterministic, so
# the results are identical every run; only the timings vary.
for run in range(num_runs):
if num_runs > 1:
print(f"\n [harness] Run {run+1} of {num_runs}")

# 7. Server side: Run aes_decryption
subprocess.run([exec_dir/"server_encrypted_aes_decryption", str(size)], check=True)
utils.log_step(7, "Encrypted aes decryption")
utils.log_size(io_dir / "ciphertext_aes_download", "Encrypted results")

# 8. Server side: Run the encrypted processing run exec_dir/server_encrypted_compute
subprocess.run([exec_dir/"server_encrypted_compute", str(size)], check=True)
utils.log_step(8, "Encrypted computation of mini workload")
utils.log_size(io_dir / "ciphertexts_download", "Encrypted results")

# 9. Client-side: decrypt
subprocess.run([exec_dir/"client_decrypt_decode_aes_decryption", str(size)], check=True)
utils.log_step(9, "Result decryption")

# 10. Client-side: post-process
subprocess.run([exec_dir/"client_postprocess_aes_decryption", str(size)], check=True)
utils.log_step(10, "Result postprocessing")

# 11. Client-side: decrypt
subprocess.run([exec_dir/"client_decrypt_decode", str(size)], check=True)
utils.log_step(11, "Result decryption")

# 12. Client-side: post-process
subprocess.run([exec_dir/"client_postprocess", str(size)], check=True)
utils.log_step(12, "Result postprocessing")

# 13. Verify aes_decryption result
aes_expected_file = params.datadir() / "expected_aes.txt"
aes_result_file = io_dir / "result_aes.txt"

if not aes_result_file.exists():
print(f"Error: Result file {aes_result_file} not found")
sys.exit(1)

subprocess.run(["python3", harness_dir/"verify_aes_decryption.py",
str(aes_expected_file), str(aes_result_file)], check=False)

# 14. Verify the final result
expected_file = params.datadir() / "max_value.txt"
workload_label = "MAX"
if mini_workload == 1:
expected_file = params.datadir() / "inner_product.txt"
workload_label = "IP"
result_file = io_dir / "result.txt"

if not result_file.exists():
print(f"Error: Result file {result_file} not found")
sys.exit(1)

subprocess.run(["python3", harness_dir/"verify_result.py",
str(expected_file), str(result_file), workload_label], check=False)

# 15. Store this run's measurements
run_path = params.measuredir() / f"results-{run+1}.json"
run_path.parent.mkdir(parents=True, exist_ok=True)
submission_report_path = io_dir / "server_reported_steps.json"
utils.save_run(run_path, submission_report_path)

print(f"\nAll steps completed for the {instance_name(size)} dataset!")

Expand Down
Loading