Intel Arc Models Runner (intel-arc-models-runner)

A purpose-built, mathematically proven, zero-bloat local LLM runner, multi-modal generative studio, and high-throughput server engineered specifically for Intel® Arc™ GPUs (discrete & integrated) and Intel® NPUs (Neural Processing Units).

---

📚 Documentation Index

To keep this landing page fast, readable, and executive, detailed technical documentation, hardware proofs, and engine manuals have been modularized into dedicated guides:

Guide / Document	Description	Key Topics
🎬 Arc MultiModal Studio	Autonomous AI Production Studio & Co-Benchmarking	Flagship scenario, 5 bottlenecks solved, Cloud Cost-Killer ROI analysis, Studio CLI & REST API.
🏗️ Architecture & Subsystems	System Architecture & Hardware Interfaces	Subsystem deep-dive, Level Zero integration, Multi-LoRA hot-swapping, Continuous batching & paged KV cache.
📦 Model Hub & Hardware Engines	25 Curated Models & Multi-Modal Engine Guides	Model matrix, Audio AI (Whisper/Kokoro), Image (SD/SDXL), Video (LTX-Video), 7-Tab Web UI, GGUF transpiler, NPU embeddings.
📊 Benchmarks Hub	Comprehensive Empirical & Quantitative Results	Entry point to all benchmarking data, testbed silicon specs, and methodology.
↳ Comparative Benchmark Matrix	Quantitative Inference Benchmark Matrix	Tables 1–7 (LLMs, 5W NPU Embeddings, Vision, Audio, Image, Video, and Table 7 Heavy Models Zero-RAM Flash-Stream) & Leviathan speculative efficiency proof.
↳ Mathematical Proofs (M59–M134)	Formal Theorems & Invariants	70 formal mathematical proofs covering Audio AI, Diffusion, Video AI, and Flash-Stream Heterogeneous Zero-RAM Execution (Proofs M121–M134).
↳ Empirical Silicon Proofs	Real Silicon Hardware Telemetry	Arc GPU vs CPU (30.8× TTFT), live sequential reclaim runs, 21-test hardware suite, GPU vs NPU breakdown, Flash-Stream memory trace ($\Delta M \le 1\text{ MB}$), and live Ollama CPU vs Arc GPU showdown.
↳ Ollama Showdown & 30-Case Audit	Head-to-Head Comparison vs Ollama	Ollama "Silent CPU Fallback" anatomy (Proof M56), output coherence (M57/M58), 30-case Triad Audit matrix, and Heavy Models Ollama OOM Collapse vs Flash-Stream Zero-RAM.
🚀 Deployment & Packaging Guide	Standalone PyPI Wheel & Docker Deployment	Production guide for pip install arc-runner, Ubuntu 24.04 Docker image, /dev/dri pass-through, and Kubernetes Helm specs.
🧪 Complete Test Coverage	Full Test Suite Roadmap & Execution	569 automated tests across 36 test modules (100% green, 0 failures), Triad testing patterns.
🗺️ Roadmap & Multi-Modal Evolution	Delivered Milestones & Packaging Horizon	15 core milestones + full Packaging Horizon delivered, 7 next-gen orchestration hurdles scorecard, PyPI & Docker targets.
📐 Visual Diagrams Guide	UML & Draw.io Diagram Catalog	13 architectural UML diagrams with embedded previews and editable draw.io links.
🔬 R&D Forensics Reports	Deep Technical R&D & Whitepapers	7 foundational research papers on Ollama defects, Intel silicon, solution blueprints, orchestration proofs, packaging math, and bare-metal flash streaming (Proofs M121–M134).
🤝 Community Benchmarks Guide	Open Source Benchmarking Call	Community guidelines for benchmarking across Intel silicon (Alchemist, Battlemage, Lunar Lake, Arrow Lake).

---

🚀 Overview

While mainstream local LLM tools like Ollama were designed with a CUDA-first and Apple Metal-first focus, intel-arc-models-runner treats Intel silicon as first-class citizens:

• Native Dual Silicon Acceleration: Run models natively on Intel Arc Discrete GPUs (A770, A750, B580), Core Ultra iGPUs (Xe-LPG, Xe2), and Intel NPUs (AI Boost NPU 3720 / NPU 4000).
• Bare-Metal Flash-Stream Heavy Model Engine: Run massive models (Qwen 2.5/3 32B & 72B) on memory-constrained devices (e.g. 1.24 GB free DRAM) with zero-RAM OOM crashes via $\mathcal{O}(1)$ layer ring buffering, asynchronous NVMe prefetch DMA, and NPU Attention / Arc GPU SwiGLU disaggregation (Proofs M121–M134)!
• World's First Client Speculative Engine: Run 1B–2B draft models on the Intel NPU (zero GPU VRAM cost, 5W power draw) while verifying tokens in parallel on the Intel Arc GPU via XMX matrix engines!
• Autonomous Multi-Modal Media Studio: Orchestrates LLM Director, Kokoro Voiceover, and SDXL / LTX-Video Visuals with zero-copy memory handoff and closed-loop pacing convergence.
• Drop-in Ollama & OpenAI Compatibility: Compatible with /api/generate, /api/chat, /api/tags, and /v1/chat/completions for instant use with Open WebUI, Continue.dev, Cursor, and LangChain.
• Zero-Bloat Standalone Runtime: Built directly on OpenVINO GenAI C++ and Level Zero runtimes; no 5 GB oneAPI SDK installation or heavy PyTorch environments required.

---

🎬 Arc MultiModal Studio Spotlight

Arc MultiModal Studio orchestrates Text, Audio, and Visual engines locally on Intel heterogeneous silicon:

User Prompt ──► Director LLM (NPU/GPU) ──► Kokoro TTS (NPU/GPU) ──► Slide Compositor / DiT (Arc GPU) ──► Muxer (.mp4)

• Cloud Cost-Killer ROI: Generates 30-second broadcast-ready educational videos locally for $0.00 vs. $0.35/video on OpenAI + ElevenLabs + Runway (pays for an Intel Arc GPU in <850 runs).
• 5 Critical Hurdles Solved: Heterogeneous Compute Splitting (8.3 GB peak VRAM), Level Zero USM Zero-Copy Bridge ($\le 50,\mu\text{s}$), Closed-Loop Pacing Convergence ($|\Delta| \le 0.05,\text{s}$), Dual-Stage AST Auto-Repair, and Fork-Join Concurrency Barriers.
• Next-Gen Features: Hardware QSV H.264 encoding, Alpha cross-dissolve, Audio-First Reactive DAG, dynamic karaoke subtitles, and autonomous VLM inspection gate.

👉 Read the full guide: Arc MultiModal Studio Documentation

---

📋 Prerequisites / Requirements

• Supported Hardware:
  • Discrete GPUs: Intel® Arc™ A770, A750, A580, A380, A310 (Alchemist) or B580, B570 (Battlemage)
  • Integrated GPUs: Intel® Core™ Ultra Series 1 (Meteor Lake Xe-LPG) or Series 2 (Lunar Lake / Arrow Lake Xe2)
  • Neural Processing Units: Intel® AI Boost NPU (NPU 3720 / NPU 4000)
• Operating System: Windows 11 / Windows 10 (64-bit), or Ubuntu 22.04+ / Linux kernel 6.6+
• Driver Stack: Intel Graphics Driver 32.0.101.6127+ or oneAPI Level Zero runtime 1.3+
• Python: Version 3.10, 3.11, 3.12, or 3.13

---

⚡ Installation / Setup

Option 1: Standalone PyPI Installation (Recommended)

Install the standalone binary distribution directly:

pip install arc-runner

Provides dual CLI commands arc-runner and intel-arc-models-runner with pre-bundled Studio Web UI and cross-platform Level Zero resolution.

Option 2: Production Docker Container (Ubuntu 24.04 with Intel Drivers)

Run directly with GPU acceleration via Docker Compose:

docker compose up -d

Or execute manual container run with direct /dev/dri render node pass-through:

docker run -d --name intel-arc-models-runner --device /dev/dri:/dev/dri -p 8000:8000 ghcr.io/techdeveloper-org/intel-arc-models-runner:latest

Option 3: 1-Click Windows Native Setup

Double-click install.bat or run in PowerShell:

powershell -ExecutionPolicy Bypass -File .\install.ps1

Option 4: Developer Editable Mode

git clone https://github.com/techdeveloper-org/intel-arc-models-runner.git
cd intel-arc-models-runner
pip install -e ".[dev]"

---

💻 Usage

1. Discover Connected Silicon & Live Telemetry

# Discover connected Intel Arc GPUs, NPUs, and CPUs
arc-runner devices

# Run real silicon integration tests with Rich terminal telemetry
arc-runner test-hardware

2. High-Throughput Daemon Server & Web UI

Launch the asynchronous HTTP gateway on port 11434 with embedded 7-Tab glassmorphic Web UI:

arc-runner serve --host 127.0.0.1 --port 11434 --device GPU

• Open http://localhost:11434/ in your browser for the Intel Arc Model Studio Web UI.
• Point Open WebUI, Continue.dev, or Cursor directly to http://localhost:11434.

3. Core CLI Quick Reference

# Pull official curated models
arc-runner pull llama-3.2-3b
arc-runner pull deepseek-r1-1.5b

# Benchmark on Intel Arc GPU or NPU with 95% confidence intervals
arc-runner benchmark llama-3.2-3b --device GPU --runs 5

# Speech-to-Text with Whisper on Arc GPU
arc-runner transcribe lecture.wav --model whisper-base --device GPU

# Text-to-Speech with Kokoro-82M on Arc GPU / NPU
arc-runner tts "Intel Arc matrix architecture accelerates local AI." --output voice.wav --voice af_heart

# Text-to-Image with Stable Diffusion / SDXL on Arc GPU
arc-runner generate-image "Cyberpunk laboratory with glowing neon circuits, 8k" --output lab.png --model lcm --steps 4

# Text-to-Video with LTX-Video on Arc GPU
arc-runner generate-video "A golden retriever puppy chasing a bouncing tennis ball, 4k" --output puppy.mp4 --model ltx-video

# Autonomous Studio: Full Video Production from natural language prompt
arc-runner studio "Quantum Physics and Wave Particle Duality" --duration 12.0 --resolution 1280x720

# Offline GGUF -> OpenVINO INT4/INT8 Transpiler
arc-runner convert ./llama-3.2-3b-Q4_0.gguf --output ./models/llama-openvino --precision int4

# NPU Vector Embeddings at 5W
arc-runner embed "What is Intel Arc GPU?" --model nomic-embed-text --device NPU

👉 For the full catalog of 25 models and detailed engine parameters, see: Model Hub & Hardware Engines Reference

---

🏗️ Architecture

flowchart TD
    subgraph Client ["Client & Tooling Layer"]
        CLI["arc-runner CLI"]
        WebUI["Studio WebUI & Open WebUI (:11434)"]
        IDE["Cursor / Continue.dev / LangChain"]
    end

    subgraph Daemon ["arc-runner Daemon (:11434)"]
        Router["Asynchronous HTTP Gateway (FastAPI)"]
        BatchPipeline["BatchExecutionPipeline & ContinuousBatcher"]
        ModelMgr["ModelManager & PagedBlockTable"]
        Router --> BatchPipeline & ModelMgr
    end

    subgraph Engines ["Pluggable Backend Engines (OpenVINO GenAI)"]
        OVEngine["OpenVINOGenAIEngine (LLMs)"]
        SpecEngine["HybridSpeculativeEngine (NPU Draft + Arc Verify)"]
        AudioEngine["AudioEngine (Whisper ASR & Kokoro TTS)"]
        ImageEngine["ImageEngine (SD 1.5, LCM, SD-Turbo, SDXL)"]
        VideoEngine["VideoEngine (LTX-Video, CogVideoX, AnimateDiff)"]
        StudioEngine["StudioDirector & MultiModal Pipeline"]
    end

    subgraph Silicon ["Physical Intel Silicon Layer (Level Zero)"]
        ArcGPU["Intel Arc GPU (Xe-LPG / Battlemage / A770)"]
        IntelNPU["Intel AI Boost NPU (3720 / 4000 @ 5W)"]
        HostCPU["Intel Core Ultra Host CPU"]
    end

    Client --> Daemon
    Daemon --> Engines
    Engines --> Silicon

Loading

👉 For detailed subsystem breakdowns, Multi-LoRA hot-swapping, and UML diagrams, see: Architecture & Subsystem Deep Dive

---

📊 Benchmark Highlights: Intel Arc vs. Ollama CPU

When Ollama runs on Intel Core Ultra systems, it silently falls back to 100% CPU execution. intel-arc-models-runner unleashes native GPU and NPU acceleration:

Workload & Model	Ollama (100% CPU Fallback)	arc-runner AI Boost NPU (5W)	arc-runner Arc Xe-LPG iGPU	arc-runner Discrete Arc A770	Hardware Advantage
Llama-3.2-1B (TTFT)	780 ms (40W)	890 ms (5.1W)	115 ms (21W)	38 ms (85W)	20.5× Faster TTFT 🚀
Llama-3.2-1B (Decode)	24.2 tok/s	19.8 tok/s (3.88 tok/J)	72.4 tok/s (84.1 Spec)	215.3 tok/s	8.9× Higher Throughput ⚡
Llama-3.2-3B (Decode)	11.4 tok/s	—	38.2 tok/s (68.5 Spec)	114.6 tok/s	10.0× Higher Throughput ⚡
all-MiniLM-L6-v2 (5W RAG)	85.2 ms (3.24 J)	18.6 ms (0.09 J)	14.2 ms (0.25 J)	—	36× More Energy Efficient! 🌱
Whisper-Base (STT)	1.944 s (RTF 0.486)	—	0.354 s (28.2× RT)	—	5.5× Faster STT ⚡
LCM Image Gen (4-Step)	2.230 s	—	0.310 s (12.9 it/s)	—	7.2× Faster Image Gen 🎨
LTX-Video Gen (16-Frame)	3.305 s (4.8 fps)	—	1.400 s (11.4 fps)	—	2.4× Faster Video Gen 🎥

👉 For the full 25-model comparison, mathematical proofs M59–M114, and 30-case Triad audit, visit: Benchmarks & Empirical Proofs Hub

---

🧪 Complete Test Coverage (533 Tests / 33 Modules)

The codebase is backed by 533 automated tests across 33 test modules with 100% passing rate, 0 failures, and 0 errors:

• Hardware Integration: 21 physical silicon tests on live Intel Core Ultra hardware.
• Multi-Modal Studio: 33 Studio Web UI triad tests, 38 slide compositor tests, and 42 next-gen orchestration tests.
• Engines & Protocols: Comprehensive coverage of OpenVINO GenAI, continuous batching, LoRA swapping, and SSE streaming.

# Run full automated test suite
python -m pytest -v --tb=short
# → 533 passed in ~24.5s

👉 For the complete test module matrix and instructions, see: Complete Test Coverage Guide

---

🗺️ Roadmap & Production Status

• ✅ Completed Core Milestones: 15 major milestones including OpenVINO continuous batching, Multi-LoRA swapping, Audio/Image/Video engines, Arc MultiModal Studio with 7-Tab Web UI, and all 7 Next-Gen Orchestration Hurdles (QSV H.264 encoding, Alpha cross-dissolve, Audio-First Reactive DAG, karaoke subtitles, audio ducking, and closed-loop VLM inspection).
• ✅ Completed Packaging Horizon:
  1. PyPI Wheel Publish: Standalone pip install arc-runner binary package distribution (arc_runner-0.1.0-py3-none-any.whl) with dual CLI aliases.
  2. Docker Container: Production Ubuntu 24.04 image with pre-baked Intel Level Zero drivers, OpenVINO runtime, non-root user execution, and /dev/dri pass-through.

👉 Read the full status & deployment specs: Roadmap & Multi-Modal Evolution and Deployment Guide

---

🤝 Contributing

intel-arc-models-runner is a community-driven, 100% open-source project. We actively welcome contributions from developers, researchers, and hardware enthusiasts worldwide!

• Calling Open-Source Contributors: Help us benchmark across Intel silicon (Arc A770/A750/B580, Lunar Lake, Arrow Lake, Meteor Lake, Xeon). All contributors receive permanent credit in our documentation Hall of Fame!
• Please review our Contributing Guidelines and Community Benchmarks Guide.

---

📄 License

Licensed under the Apache License, Version 2.0.