diff --git a/README.md b/README.md
index 97d05ce6..ad3ecc25 100644
--- a/README.md
+++ b/README.md
@@ -1,508 +1,310 @@
-# GPULlama3.java powered by TornadoVM [](https://github.com/beehive-lab/GPULlama3.java/actions/workflows/build-and-run.yml) [](https://central.sonatype.com/artifact/io.github.beehive-lab/gpu-llama3)
+# GPULlama3.java โ LLM inference & serving for the JVM, on any GPU
-
-
-
+[](https://github.com/beehive-lab/GPULlama3.java/actions/workflows/build-and-run.yml)
+[](https://central.sonatype.com/artifact/io.github.beehive-lab/gpu-llama3)
+
+
[](https://docs.langchain4j.dev/)
-
-
-[](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl)
-[](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-ptx)
-[](https://deepwiki.com/beehive-lab/GPULlama3.java)
+
+
+
+[](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl)
+[](https://deepwiki.com/beehive-lab/GPULlama3.java)
+[](LICENSE)
-----------
+
|
-
-Llama3 models written in native Java automatically accelerated on GPUs with TornadoVM.
-Runs Llama3 inference efficiently using TornadoVM's GPU acceleration.
-
-Currently, supports Llama3, Mistral, Devstral 2, Qwen2.5, Qwen3, Phi-3, IBM Granite 3.2+ and IBM Granite 4.0 models in the GGUF format.
-Also, it is used as GPU inference engine in
-Quarkus
-and
-LangChain4J.
-
-Builds on Llama3.java by Alfonsoยฒ Peterssen.
-Previous integration of TornadoVM and Llama2 it can be found in llama2.tornadovm.
- |
-
-
+
------------
-##  Integration with LangChain4j
+### Think vLLM โ but pure Java, and it runs on **any** GPU.
-Since **LangChain4j v1.7.1**, `GPULlama3.java` is officially supported as a **model provider**.
-This means you can directly use *GPULlama3.java* inside your LangChain4j applications without extra glue code, just run on your GPU.
+**GPULlama3.java** is a JVM-native LLM inference and serving engine. You write and ship plain Java; [**TornadoVM**](https://github.com/beehive-lab/TornadoVM) JIT-compiles the hot transformer kernels to **CUDA, PTX, OpenCL, or Apple Metal** at runtime โ no JNI glue, no second toolchain, no native rebuild per GPU.
-๐ Learn more: [LangChain4j Documentation](https://docs.langchain4j.dev/)
+One `.jar` runs the same model on **NVIDIA, Intel, AMD, and Apple Silicon**, from a laptop to an RTX 5090.
-[Example agentic workflows with GPULlama3.java + LangChain4j ๐](https://github.com/mikepapadim/devoxx25-demo-gpullama3-langchain4j/tree/main)
+Serve it behind an **OpenAI-compatible API**, embed it in **LangChain4j** or **Quarkus**, or run it from the CLI in one line.
-How to use:
-```java
-GPULlama3ChatModel model = GPULlama3ChatModel.builder()
- .modelPath(modelPath)
- .temperature(0.9) // more creative
- .topP(0.9) // more variety
- .maxTokens(2048)
- .onGPU(Boolean.TRUE) // if false, runs on CPU though a lightweight implementation of llama3.java
- .build();
-```
------------
-#### **[Interactive-mode]** Running on a RTX 5090 with nvtop on bottom to track GPU utilization and memory usage.
+ |
+
+
-
+Builds on [Llama3.java](https://github.com/mukel/llama3.java) by [Alfonsoยฒ Peterssen](https://github.com/mukel). Earlier Llama2 work: [llama2.tornadovm](https://github.com/mikepapadim/llama2.tornadovm.java).
-----------
-## Setup & Configuration
-
-### Prerequisites
-
-Ensure you have the following installed and configured:
+## Why GPULlama3.java
-- **Java 21**: Required for Vector API support & TornadoVM.
-- [TornadoVM](https://github.com/beehive-lab/TornadoVM) with OpenCL or PTX backends.
-- GCC/G++ 13 or newer: Required to build and run TornadoVM native components.
+- ๐ฆ **Pure Java, all the way down.** Transformer kernels are written in Java and accelerated by TornadoVM โ no CUDA C, no hand-written JNI. Debug and build with the toolchain you already have.
+- ๐ **Write once, run on any GPU.** NVIDIA (CUDA / PTX), Intel & AMD (OpenCL), Apple Silicon (Metal). Switch backends with a flag, not a rebuild.
+- ๐ **Drop-in for the Java AI stack.** Official [LangChain4j](https://docs.langchain4j.dev/integrations/language-models/gpullama3-java) provider (since v1.7.1) and [Quarkus](https://docs.quarkiverse.io/quarkus-langchain4j/dev/gpullama3-chat-model.html) inference engine.
+- โก **Built to serve.** OpenAI-compatible HTTP server, batched decode, and on-device sampling (see [Serving](#-serving-openai-compatible-preview)).
+- ๐ฆ **Many models, one runtime.** Llama 3, Mistral, Qwen 2.5 / Qwen 3, Phi-3, IBM Granite 3.3 / 4.0, DeepSeek-R1-Distill โ all in GGUF.
-### Install, Build, and Run
-
-```bash
-# Clone the repository with all submodules
-git clone https://github.com/beehive-lab/GPULlama3.java.git
-```
-
-#### Install the TornadoVM SDK on Linux or macOS
-
-Ensure that your JAVA_HOME points to a supported JDK before using the SDK. Download an SDK package matching your OS, architecture, and accelerator backend (opencl, ptx).
-TornadoVM is distributed through our [**official website**](https://www.tornadovm.org/downloads) and **SDKMAN!**. Install a version that matches your OS, architecture, and accelerator backend.
-
-All TornadoVM SDKs are available on the [SDKMAN! TornadoVM page](https://sdkman.io/sdks/tornadovm/).
+-----------
-#### SDKMAN! Installation (Recommended)
+## โฑ๏ธ Quickstart (60 seconds)
-##### Install SDKMAN! if not installed already
-```bash
-curl -s "https://get.sdkman.io" | bash
-source "$HOME/.sdkman/bin/sdkman-init.sh"
-sdk version
-```
-##### Install TornadoVM via SDKMAN!
```bash
+# 1. Install a TornadoVM SDK (bundles the GPU runtime)
+curl -s "https://get.sdkman.io" | bash && source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install tornadovm
-```
+tornado --devices # confirm your GPU is listed
-#### Verify TornadoVM is Installed Correctly
-```bash
-tornado --devices
+# 2. Run a model on the GPU โ no build required, via JBang
+jbang gpullama3@beehive-lab -m beehive-llama-3.2-1b-instruct-fp16.gguf -p "Explain GPU acceleration in one sentence."
```
-----------
-
-### TornadoVM-Accelerated Inference Performance and Optimization Status
-We are at the early stages of Java entering the AI world with features added to the JVM that enable faster execution such as GPU acceleration, Vector acceleration, high-performance access to off-heap memory and others.
+Grab a ready-to-run model from the [Hugging Face collections](#-model-collections) below.
+-----------
-| Vendor / Backend | Hardware | Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | Optimizations |
-|:----------------------------:|:------------:|:---------------------:|:---------------------:|:-------------:|
-| | | **FP16** | **FP16** | **Support** |
-| **NVIDIA / OpenCL-PTX** | RTX 3070 | 66 tokens/s | 55.46 tokens/s | โ
|
-| | RTX 4090 | 86.11 tokens/s | 75.32 tokens/s | โ
|
-| | RTX 5090 | 117.65 tokens/s | 112.68 tokens/s | โ
|
-| | L4 Tensor | 52.96 tokens/s | 22.68 tokens/s | โ
|
-| **Intel / OpenCL** | Arc A770 | 15.65 tokens/s | 7.02 tokens/s | (WIP) |
-| **Apple Silicon / OpenCL** | M3 Pro | 14.04 tokens/s | 6.78 tokens/s | (WIP) |
-| | M4 Pro | 16.77 tokens/s | 8.56 tokens/s | (WIP) |
-| **AMD / OpenCL** | Radeon RX | (WIP) | (WIP) | (WIP) |
+## ๐งฉ Serving: OpenAI-compatible (preview)
-##### Apple Silicon Support
+GPULlama3.java is growing into a **serving engine** โ the vLLM-style path for the JVM. These land through active pull requests; try them from the linked branches today:
-TornadoVM 4.0 includes a native [Metal](https://developer.apple.com/metal/) backend, enabling GPU-accelerated inference on Apple Silicon.
+- ๐ **OpenAI-compatible server** โ `llama-tornado --server` exposes `/v1/chat/completions` and `/v1/completions` with streaming and zero external dependencies. Point any OpenAI client at `localhost`. ([PR #135](https://github.com/beehive-lab/GPULlama3.java/pull/135))
+- ๐งฎ **On-device greedy sampling** โ argmax on the GPU keeps logits device-side, cutting deviceโhost traffic by ~500ร per token. ([PR #134](https://github.com/beehive-lab/GPULlama3.java/pull/134))
+- ๐ **Static batched decode** โ B independent sequences per step for up to **41ร aggregate throughput** (Llama & Qwen3). ([PR #129](https://github.com/beehive-lab/GPULlama3.java/pull/129))
+- ๐ฏ **Tensor-core (MMA) batch prefill** on the CUDA backend, FP16 & Q8_0. ([PR #127](https://github.com/beehive-lab/GPULlama3.java/pull/127))
+- ๐ **llama-bench-style benchmarking** โ `llama-tornado --bench` reports a pp/tg matrix with avgยฑstddev in md/csv/json. ([PR #133](https://github.com/beehive-lab/GPULlama3.java/pull/133))
-----------
-## ๐ฆ Maven Dependency
-You can add **GPULlama3.java** directly to your Maven project by including the following dependency in your `pom.xml`:
+## LangChain4j & Quarkus
-**JDK 21:**
-```xml
-
- io.github.beehive-lab
- gpu-llama3
- 0.5.0
-
+Since **LangChain4j v1.7.1**, `GPULlama3.java` is an officially supported **model provider** โ no glue code, GPU-accelerated out of the box.
+
+```java
+GPULlama3ChatModel model = GPULlama3ChatModel.builder()
+ .modelPath(modelPath)
+ .temperature(0.9) // more creative
+ .topP(0.9) // more variety
+ .maxTokens(2048)
+ .onGPU(Boolean.TRUE) // false โ lightweight CPU llama3.java
+ .build();
```
-**JDK 25:**
+๐ [LangChain4j docs](https://docs.langchain4j.dev/) ยท ๐ [Agentic workflow demo](https://github.com/mikepapadim/devoxx25-demo-gpullama3-langchain4j/tree/main)
+
+### ๐ฆ Maven
+
```xml
+
io.github.beehive-lab
gpu-llama3
- 0.5.0-jdk25
+ 0.5.0
-```
-## โ Integration with Your Java Codebase or Tools
-
-To integrate it into your codebase or IDE (e.g., IntelliJ) or custom build system (like IntelliJ, Maven, or Gradle), use the `--show-command` flag.
-This flag shows the exact Java command with all JVM flags that are being invoked under the hood to enable seamless execution on GPUs with TornadoVM.
-Hence, it makes it simple to replicate or embed the invoked flags in any external tool or codebase.
-
-```bash
-llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "tell me a joke" --show-command
+
```
-
-๐ Click to see the JVM configuration
-
-```java
-/home/mikepapadim/.sdkman/candidates/java/current/bin/java \
- -server \
- -XX:+UnlockExperimentalVMOptions \
- -XX:+EnableJVMCI \
- -Xms20g -Xmx20g \
- --enable-preview \
- -Djava.library.path=/home/mikepapadim/manchester/TornadoVM/bin/sdk/lib \
- -Djdk.module.showModuleResolution=false \
- --module-path .:/home/mikepapadim/manchester/TornadoVM/bin/sdk/share/java/tornado \
- -Dtornado.load.api.implementation=uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph \
- -Dtornado.load.runtime.implementation=uk.ac.manchester.tornado.runtime.TornadoCoreRuntime \
- -Dtornado.load.tornado.implementation=uk.ac.manchester.tornado.runtime.common.Tornado \
- -Dtornado.load.annotation.implementation=uk.ac.manchester.tornado.annotation.ASMClassVisitor \
- -Dtornado.load.annotation.parallel=uk.ac.manchester.tornado.api.annotations.Parallel \
- -Dtornado.tvm.maxbytecodesize=65536 \
- -Duse.tornadovm=true \
- -Dtornado.threadInfo=false \
- -Dtornado.debug=false \
- -Dtornado.fullDebug=false \
- -Dtornado.printKernel=false \
- -Dtornado.print.bytecodes=false \
- -Dtornado.device.memory=7GB \
- -Dtornado.profiler=false \
- -Dtornado.log.profiler=false \
- -Dtornado.profiler.dump.dir=/home/mikepapadim/repos/gpu-llama3.java/prof.json \
- -Dtornado.enable.fastMathOptimizations=true \
- -Dtornado.enable.mathOptimizations=false \
- -Dtornado.enable.nativeFunctions=fast \
- -Dtornado.loop.interchange=true \
- -Dtornado.eventpool.maxwaitevents=32000 \
- "-Dtornado.opencl.compiler.flags=-cl-denorms-are-zero -cl-no-signed-zeros -cl-finite-math-only" \
- --upgrade-module-path /home/mikepapadim/manchester/TornadoVM/bin/sdk/share/java/graalJars \
- @/home/mikepapadim/manchester/TornadoVM/bin/sdk/etc/exportLists/common-exports \
- @/home/mikepapadim/manchester/TornadoVM/bin/sdk/etc/exportLists/opencl-exports \
- --add-modules ALL-SYSTEM,tornado.runtime,tornado.annotation,tornado.drivers.common,tornado.drivers.opencl \
- -cp /home/mikepapadim/repos/gpu-llama3.java/target/gpu-llama3-1.0-SNAPSHOT.jar \
- org.beehive.gpullama3.LlamaApp \
- -m beehive-llama-3.2-1b-instruct-fp16.gguf \
- --temperature 0.1 \
- --top-p 0.95 \
- --seed 1746903566 \
- --max-tokens 512 \
- --stream true \
- --echo false \
- -p "tell me a joke" \
- --instruct
-```
-
-
-
-----------
-The above model can we swapped with one of the other models, such as `beehive-llama-3.2-3b-instruct-fp16.gguf` or `beehive-llama-3.2-8b-instruct-fp16.gguf`, depending on your needs.
-Check models below.
+#### **[Interactive mode]** โ RTX 5090, with `nvtop` tracking GPU utilization and memory
------------
+
-## ๐ Running with JBang (Pure Java CLI)
+-----------
-You can run llama-tornado as a pure Java script using [JBang](https://www.jbang.dev/) without building or installing anything. This provides a simple, script-like experience similar to [Jlama's CLI](https://github.com/tjake/Jlama).
+## ๐ ๏ธ Install & build
-### Prerequisites for JBang
+### Prerequisites
-1. **Install JBang**: Follow the [JBang installation guide](https://www.jbang.dev/download/)
-2. **TornadoVM SDK**: You still need TornadoVM installed and `TORNADOVM_HOME` environment variable set (see Setup section above)
+- **Java 21** โ required for the Vector API & TornadoVM (Java 25 supported via the `-jdk25` artifact / `llamaTornado` script).
+- **[TornadoVM](https://github.com/beehive-lab/TornadoVM)** with an OpenCL, PTX, or CUDA backend. The `--cuda` backend needs a TornadoVM build with the CUDA backend ([PR #861](https://github.com/beehive-lab/TornadoVM/pull/861)); this project builds against TornadoVM `5.0.0-jdk21-dev`.
+- **GCC/G++ 13+** โ to build TornadoVM's native components.
-### Quick Start with JBang
+### Get TornadoVM (SDKMAN!, recommended)
-Use from catalog:
+TornadoVM is distributed via the [official website](https://www.tornadovm.org/downloads) and [SDKMAN!](https://sdkman.io/sdks/tornadovm/). Pick a package matching your OS, architecture, and backend (opencl, ptx).
```bash
-# Install JBang (if not already installed)
-curl -Ls https://sh.jbang.dev | bash -s - app setup
-
-# Run GPULlama3.java CLI
-jbang gpullama3@beehive-lab -m model.gguf -p "Tell me a joke"
-
-# Or install it as a command
-jbang app install gpullama3@beehive-lab
-gpullama3 -m model.gguf -p "Hello!"
+sdk install tornadovm
+tornado --devices # verify
```
-or the local:
-```bash
-# Basic usage - interactive chat mode
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf --interactive
-# Single instruction mode
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf -p "Explain quantum computing"
+### Clone this repo
-# With TornadoVM GPU acceleration
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf \
- -p "Tell me a joke" --use-tornadovm true
-
-# Custom generation parameters
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf \
- -p "Write a short story" \
- --temperature 0.7 \
- --top-p 0.9 \
- --max-tokens 512
+```bash
+git clone https://github.com/beehive-lab/GPULlama3.java.git
```
-----------
-## Collection of Tested Models
-
-### Llama3.2 Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/llama3-gpullama3java)
-
-### IBM Granite 4.0 Collection
-[https://huggingface.co/collections/beehive-lab/granite-40-language-models-gpullama3java](https://huggingface.co/collections/beehive-lab/granite-40-language-models-gpullama3java)
+## โถ๏ธ Running the CLI
+Use the `llama-tornado` script with `--gpu`. Pick a backend with `--opencl`, `--ptx`, `--cuda` (NVIDIA), or `--metal` (Apple Silicon).
-### IBM Granite 3.3 Collection
-[https://huggingface.co/collections/beehive-lab/granite-33-language-models-gpullama3java](https://huggingface.co/collections/beehive-lab/granite-33-language-models-gpullama3java)
-
-### Qwen 2.5 Collection
-[https://huggingface.co/collections/beehive-lab/qwen-25-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-25-gpullama3java)
-
-### Qwen 3 Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-3-gpullama3java)
-
-### Phi-3 Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/phi-3-gpullama3java)
-
-### Mistral Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/mistral-gpullama3java)
-
-### DeepSeek-R1-Distill-Qwen Collection
-[https://huggingface.co/collections/beehive-lab/deepseek-r1-distill-qwen-gpullama3java](https://huggingface.co/collections/beehive-lab/deepseek-r1-distill-qwen-gpullama3java)
-
------------
-
-## Running `llama-tornado`
+```bash
+# Basic GPU inference (OpenCL)
+./llama-tornado --gpu --verbose-init --opencl \
+ --model beehive-llama-3.2-1b-instruct-fp16.gguf \
+ --prompt "Explain the benefits of GPU acceleration."
+
+# CUDA backend
+./llama-tornado --gpu --cuda \
+ --model beehive-llama-3.2-1b-instruct-fp16.gguf \
+ --prompt "Explain the benefits of GPU acceleration."
+```
-To execute Llama3, or Mistral models with TornadoVM on GPUs use the `llama-tornado` script with the `--gpu` flag.
+Swap in any tested model โ e.g. `beehive-llama-3.2-3b-instruct-fp16.gguf` or `...-8b-...`.
-### Usage Examples
+### `llamaTornado` โ zero-dependency Java 25 script
-#### Basic Inference
-Run a model with a text prompt:
+A single-file Java 25 launcher that replaces the Python script (needs `java 25+` on your PATH):
```bash
-./llama-tornado --gpu --verbose-init --opencl --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "Explain the benefits of GPU acceleration."
+./llamaTornado --gpu --verbose-init --metal \
+ --model Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"
```
-#### GPU Execution (FP16 Model)
-Enable GPU acceleration with Q8_0 quantization:
-```bash
-./llama-tornado --gpu --verbose-init --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "tell me a joke"
-```
+### ๐ JBang โ run without building
-#### Running with `llamaTornado` (Java 25 single-file script)
-
-`llamaTornado` is a zero-dependency Java 25 single-file script that replaces the Python launcher. It requires `java 25+` on your PATH:
+Script-like startup ร la [Jlama](https://github.com/tjake/Jlama), powered by [JBang](https://www.jbang.dev/):
```bash
-./llamaTornado --gpu --verbose-init --metal --model /Users/abien/work/workspaces/llms/Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"
-```
-
------------
+curl -Ls https://sh.jbang.dev | bash -s - app setup
-## ๐ณ Docker
+# From the catalog
+jbang gpullama3@beehive-lab -m model.gguf -p "Tell me a joke"
+jbang app install gpullama3@beehive-lab && gpullama3 -m model.gguf -p "Hello!"
-You can run `GPULlama3.java` fully containerized with GPU acceleration enabled via **OpenCL** or **PTX** using pre-built Docker images.
-More information as well as examples to run with the containers are available at [docker-gpullama3.java](https://github.com/beehive-lab/docker-gpullama3.java).
+# Or the local script, interactive
+jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf --interactive
+```
-### ๐ฆ Available Docker Images
+### ๐ณ Docker
-| Backend | Docker Image | Pull Command |
-|--------|---------------|---------------|
-| **OpenCL** | [`beehivelab/gpullama3.java-nvidia-openjdk-opencl`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl) | `docker pull beehivelab/gpullama3.java-nvidia-openjdk-opencl` |
-| **PTX (CUDA)** | [`beehivelab/gpullama3.java-nvidia-openjdk-ptx`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-ptx) | `docker pull beehivelab/gpullama3.java-nvidia-openjdk-ptx` |
+Fully containerized GPU inference via pre-built images ([docker-gpullama3.java](https://github.com/beehive-lab/docker-gpullama3.java)):
-#### Example (OpenCL)
+| Backend | Image |
+|---------|-------|
+| **OpenCL** | [`beehivelab/gpullama3.java-nvidia-openjdk-opencl`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl) |
+| **PTX (CUDA)** | [`beehivelab/gpullama3.java-nvidia-openjdk-ptx`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-ptx) |
```bash
-docker run --rm -it --gpus all \
- -v "$PWD":/data \
+docker run --rm -it --gpus all -v "$PWD":/data \
beehivelab/gpullama3.java-nvidia-openjdk-opencl \
/gpullama3/GPULlama3.java/llama-tornado \
- --gpu --verbose-init \
- --opencl \
- --model /data/Llama-3.2-1B-Instruct.FP16.gguf \
- --prompt "Tell me a joke"
+ --gpu --verbose-init --opencl \
+ --model /data/Llama-3.2-1B-Instruct.FP16.gguf --prompt "Tell me a joke"
```
+
-----------
-## Troubleshooting GPU Memory Issues
+## ๐ค Model collections
-### Out of Memory Error
+GGUF models, ready to download:
-You may encounter an out-of-memory error like:
-```
-Exception in thread "main" uk.ac.manchester.tornado.api.exceptions.TornadoOutOfMemoryException: Unable to allocate 100663320 bytes of memory.
-To increase the maximum device memory, use -Dtornado.device.memory=GB
-```
+| Family | Collection |
+|--------|-----------|
+| Llama 3.2 | [llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/llama3-gpullama3java) |
+| IBM Granite 4.0 | [granite-40-language-models](https://huggingface.co/collections/beehive-lab/granite-40-language-models-gpullama3java) |
+| IBM Granite 3.3 | [granite-33-language-models](https://huggingface.co/collections/beehive-lab/granite-33-language-models-gpullama3java) |
+| Qwen 2.5 | [qwen-25-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-25-gpullama3java) |
+| Qwen 3 | [qwen-3-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-3-gpullama3java) |
+| Phi-3 | [phi-3-gpullama3java](https://huggingface.co/collections/beehive-lab/phi-3-gpullama3java) |
+| Mistral | [mistral-gpullama3java](https://huggingface.co/collections/beehive-lab/mistral-gpullama3java) |
+| DeepSeek-R1-Distill-Qwen | [deepseek-r1-distill-qwen](https://huggingface.co/collections/beehive-lab/deepseek-r1-distill-qwen-gpullama3java) |
-This indicates that the default GPU memory allocation (7GB) is insufficient for your model.
+Formats: GGUF ยท FP16 (full), Q8_0 & Q4_0 (partial).
-### Solution
+-----------
-First, check your GPU specifications. If your GPU has high memory capacity, you can increase the GPU memory allocation using the `--gpu-memory` flag:
+## ๐พ GPU memory
-```bash
-# For 3B models, try increasing to 15GB
-./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-fp16.gguf --prompt "Tell me a joke" --gpu-memory 15GB
+Default device allocation is **7GB**. Larger models need more โ raise it with `--gpu-memory`:
-# For 8B models, you may need even more (20GB or higher)
-./llama-tornado --gpu --model beehive-llama-3.2-8b-instruct-fp16.gguf --prompt "Tell me a joke" --gpu-memory 20GB
-```
+| Model size | Recommended | Flag |
+|------------|-------------|------|
+| 1B | 7GB (default) | โ |
+| 3โ7B | 15GB | `--gpu-memory 15GB` |
+| 8B+ | 20GB+ | `--gpu-memory 20GB` |
-### GPU Memory Requirements by Model Size
-
-| Model Size | Recommended GPU Memory |
-|-------------|------------------------|
-| 1B models | 7GB (default) |
-| 3-7B models | 15GB |
-| 8B models | 20GB+ |
+```bash
+./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-fp16.gguf \
+ --prompt "Tell me a joke" --gpu-memory 15GB
+```
-**Note**: If you still encounter memory issues, try:
+Still out of memory? Use Q4_0 instead of Q8_0, or close other GPU apps. The error to look for:
-1. Using Q4_0 instead of Q8_0 quantization (requires less memory).
-2. Closing other GPU-intensive applications in your system.
+```
+TornadoOutOfMemoryException: Unable to allocate ... bytes of memory.
+To increase the maximum device memory, use -Dtornado.device.memory=GB
+```
-----------
-## Command Line Options
+## ๐ง Embed in your own tools
-Supported command-line options include:
+`--show-command` prints the exact Java + JVM invocation used under the hood, so you can replicate it in IntelliJ, Maven, Gradle, or any launcher:
```bash
-cmd โ llama-tornado --help
-usage: llama-tornado [-h] --model MODEL_PATH [--prompt PROMPT] [-sp SYSTEM_PROMPT] [--temperature TEMPERATURE] [--top-p TOP_P] [--seed SEED] [-n MAX_TOKENS]
- [--stream STREAM] [--echo ECHO] [-i] [--instruct] [--gpu] [--opencl] [--ptx] [--gpu-memory GPU_MEMORY] [--heap-min HEAP_MIN] [--heap-max HEAP_MAX]
- [--debug] [--profiler] [--profiler-dump-dir PROFILER_DUMP_DIR] [--print-bytecodes] [--print-threads] [--print-kernel] [--full-dump]
- [--show-command] [--execute-after-show] [--opencl-flags OPENCL_FLAGS] [--max-wait-events MAX_WAIT_EVENTS] [--verbose]
-
-GPU-accelerated LLaMA.java model runner using TornadoVM
-
-options:
- -h, --help show this help message and exit
- --model MODEL_PATH Path to the LLaMA model file (e.g., beehive-llama-3.2-8b-instruct-fp16.gguf) (default: None)
-
-LLaMA Configuration:
- --prompt PROMPT Input prompt for the model (default: None)
- -sp SYSTEM_PROMPT, --system-prompt SYSTEM_PROMPT
- System prompt for the model (default: None)
- --temperature TEMPERATURE
- Sampling temperature (0.0 to 2.0) (default: 0.1)
- --top-p TOP_P Top-p sampling parameter (default: 0.95)
- --seed SEED Random seed (default: current timestamp) (default: None)
- -n MAX_TOKENS, --max-tokens MAX_TOKENS
- Maximum number of tokens to generate (default: 512)
- --stream STREAM Enable streaming output (default: True)
- --echo ECHO Echo the input prompt (default: False)
- --suffix SUFFIX Suffix for fill-in-the-middle request (Codestral) (default: None)
-
-Mode Selection:
- -i, --interactive Run in interactive/chat mode (default: False)
- --instruct Run in instruction mode (default) (default: True)
-
-Hardware Configuration:
- --gpu Enable GPU acceleration (default: False)
- --opencl Use OpenCL backend (default) (default: None)
- --ptx Use PTX/CUDA backend (default: None)
- --gpu-memory GPU_MEMORY
- GPU memory allocation (default: 7GB)
- --heap-min HEAP_MIN Minimum JVM heap size (default: 20g)
- --heap-max HEAP_MAX Maximum JVM heap size (default: 20g)
-
-Debug and Profiling:
- --debug Enable debug output (default: False)
- --profiler Enable TornadoVM profiler (default: False)
- --profiler-dump-dir PROFILER_DUMP_DIR
- Directory for profiler output (default: /home/mikepapadim/repos/gpu-llama3.java/prof.json)
-
-TornadoVM Execution Verbose:
- --print-bytecodes Print bytecodes (tornado.print.bytecodes=true) (default: False)
- --print-threads Print thread information (tornado.threadInfo=true) (default: False)
- --print-kernel Print kernel information (tornado.printKernel=true) (default: False)
- --full-dump Enable full debug dump (tornado.fullDebug=true) (default: False)
- --verbose-init Enable timers for TornadoVM initialization (llama.EnableTimingForTornadoVMInit=true) (default: False)
-
-Command Display Options:
- --show-command Display the full Java command that will be executed (default: False)
- --execute-after-show Execute the command after showing it (use with --show-command) (default: False)
-
-Advanced Options:
- --opencl-flags OPENCL_FLAGS
- OpenCL compiler flags (default: -cl-denorms-are-zero -cl-no-signed-zeros -cl-finite-math-only)
- --max-wait-events MAX_WAIT_EVENTS
- Maximum wait events for TornadoVM event pool (default: 32000)
- --verbose, -v Verbose output (default: False)
-
+llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf \
+ --prompt "tell me a joke" --show-command
```
-## Debug & Profiling Options
-View TornadoVM's internal behavior:
-```bash
-# Print thread information during execution
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-threads
-
-# Show bytecode compilation details
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-bytecodes
+
+๐ Full command-line options (llama-tornado --help)
-# Display generated GPU kernel code
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-kernel
+```
+usage: llama-tornado [-h] --model MODEL_PATH [--prompt PROMPT] [-sp SYSTEM_PROMPT]
+ [--temperature TEMPERATURE] [--top-p TOP_P] [--seed SEED] [-n MAX_TOKENS]
+ [--stream STREAM] [--echo ECHO] [-i] [--instruct]
+ [--gpu] [--opencl] [--ptx] [--cuda] [--metal]
+ [--gpu-memory GPU_MEMORY] [--heap-min HEAP_MIN] [--heap-max HEAP_MAX]
+ [--debug] [--profiler] [--profiler-dump-dir DIR]
+ [--print-bytecodes] [--print-threads] [--print-kernel] [--full-dump]
+ [--show-command] [--execute-after-show]
+ [--opencl-flags FLAGS] [--max-wait-events N] [--verbose]
+
+LLaMA Configuration: --prompt, -sp/--system-prompt, --temperature (0.0โ2.0, default 0.1),
+ --top-p (default 0.95), --seed, -n/--max-tokens (default 512),
+ --stream (default True), --echo (default False), --suffix (FIM/Codestral)
+Mode Selection: -i/--interactive, --instruct (default)
+Hardware: --gpu, --opencl (default), --ptx, --cuda, --metal,
+ --gpu-memory (default 7GB), --heap-min/--heap-max (default 20g)
+Debug & Profiling: --debug, --profiler, --profiler-dump-dir,
+ --print-bytecodes, --print-threads, --print-kernel, --full-dump, --verbose-init
+Command Display: --show-command, --execute-after-show
+Advanced: --opencl-flags (default: -cl-denorms-are-zero -cl-no-signed-zeros
+ -cl-finite-math-only), --max-wait-events (default 32000), --verbose/-v
+```
-# Enable full debug output with all details
-./llama-tornado --gpu --model model.gguf --prompt "..." --debug --full-dump
+
-# Combine debug options
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-threads --print-bytecodes --print-kernel
+```bash
+# Peek at what TornadoVM is doing
+./llama-tornado --gpu --model model.gguf --prompt "..." --print-kernel # generated GPU kernel
+./llama-tornado --gpu --model model.gguf --prompt "..." --print-bytecodes # TornadoVM bytecodes
+./llama-tornado --gpu --model model.gguf --prompt "..." --debug --full-dump # everything
```
-## Current Features & Roadmap
+-----------
- - **Support for GGUF format models** with full FP16 and partial support for Q8_0 and Q4_0 quantization.
- - **Instruction-following and chat modes** for various use cases.
- - **Interactive CLI** with `--interactive` and `--instruct` modes.
- - **Flexible backend switching** - choose OpenCL or PTX at runtime (need to build TornadoVM with both enabled).
- - **Cross-platform compatibility**:
- - โ
NVIDIA GPUs (OpenCL & PTX )
- - โ
Intel GPUs (OpenCL)
- - โ
Apple GPUs (OpenCL)
+## ๐บ๏ธ Features & roadmap
-Click [here](https://github.com/beehive-lab/GPULlama3.java/tree/main/docs/TORNADOVM_TRANSFORMER_OPTIMIZATIONS.md) to view a more detailed list of the transformer optimizations implemented in TornadoVM.
+- โ
**GGUF models** โ full FP16, partial Q8_0 / Q4_0.
+- โ
**Chat, instruction, and interactive** modes (`--interactive`, `--instruct`).
+- โ
**Runtime backend switching** โ OpenCL / PTX / CUDA / Metal (build TornadoVM with the backends you need).
+- โ
**Cross-platform**: NVIDIA (OpenCL ยท PTX ยท CUDA), Intel (OpenCL), Apple (OpenCL ยท Metal).
+- ๐งฉ **Serving** โ OpenAI-compatible API, batched decode, on-device sampling (preview; see [Serving](#-serving-openai-compatible-preview)).
-Click [here](https://github.com/beehive-lab/GPULlama3.java/tree/main/docs/GPULlama3_ROADMAP.md) to see the roadmap of the project.
+๐ [Transformer optimizations in TornadoVM](docs/TORNADOVM_TRANSFORMER_OPTIMIZATIONS.md) ยท ๐งญ [Project roadmap](docs/GPULlama3_ROADMAP.md)
-----------
-## Acknowledgments
+## ๐ Acknowledgments
-This work is partially funded by the following EU & UKRI grants (most recent first):
-
-- EU Horizon Europe & UKRI [AERO 101092850](https://aero-project.eu/).
-- EU Horizon Europe & UKRI [P2CODE 101093069](https://p2code-project.eu/).
-- EU Horizon Europe & UKRI [ENCRYPT 101070670](https://encrypt-project.eu).
-- EU Horizon Europe & UKRI [TANGO 101070052](https://tango-project.eu).
-
------------
+Partially funded by EU Horizon Europe & UKRI grants (most recent first):
+[AERO 101092850](https://aero-project.eu/) ยท [P2CODE 101093069](https://p2code-project.eu/) ยท [ENCRYPT 101070670](https://encrypt-project.eu) ยท [TANGO 101070052](https://tango-project.eu).
## License
-MIT
+[MIT](LICENSE)