Skip to content

Add Docker container capability - #295

Open
recrudesce wants to merge 2 commits into
FlashML-org:mainfrom
recrudesce:docker
Open

Add Docker container capability#295
recrudesce wants to merge 2 commits into
FlashML-org:mainfrom
recrudesce:docker

Conversation

@recrudesce

Copy link
Copy Markdown

Summary of Changes

Adds complete Docker and Docker Compose support for deploying FreeToken with GPU acceleration and native Hugging Face model integration.


What's Changed

  1. Dockerfile:

    • Uses nvidia/cuda:13.3.1-devel-ubuntu26.04 as the base image to provide nvcc and CUDA headers needed for FreeToken's C++ extensions and JIT kernel compilation.
    • Sets up build environment paths (CUDA_HOME, CPATH, LIBRARY_PATH, LD_LIBRARY_PATH) to ensure compilation headers (e.g. cuda_runtime_api.h) are resolved without errors.
    • Manages dependencies using uv with a pinned Python 3.11 virtual environment to adhere to PEP 668 and avoid ABI mismatches with PyTorch extension builds.
    • Installs freetoken[accel] with pre-installed build toolchain (torch, setuptools, wheel, ninja).
    • Sets ft serve as the container entrypoint exposing port 1919.
  2. docker-compose.yml:

    • Configures NVIDIA GPU resource reservations via the NVIDIA Container Toolkit.
    • Sets ipc: host to allocate sufficient shared memory for tensor parallelism and KV cache operations.
    • Binds ~/.cache/huggingface to persist downloaded model weights across container restarts.
    • Parameterizes the model identifier via the MODEL_NAME environment variable.
  3. Configuration & Documentation:

    • Added DOCKER_README.md containing end-to-end instructions for building, configuring, running via Compose or CLI, and verifying inference via OpenAI-compatible endpoints.

Verification

  • Docker image builds successfully without header or ABI compilation errors.
  • NVIDIA GPU passthrough verified inside the container runtime.
  • Container boots ft serve and pulls model weights directly from Hugging Face into the mounted cache volume.
  • Verified OpenAI-compatible completions endpoint on http://127.0.0.1:1919/v1/chat/completions.

@recrudesce recrudesce mentioned this pull request Aug 30, 2026
@benwilson

Copy link
Copy Markdown

Two install traps worth guarding against in this Dockerfile, both hit on a stock pytorch/pytorch:2.11.0-cuda13.0-cudnn9-devel base before we got a working container (your nvidia/cuda + uv base may already sidestep them, but they cost us two launches):

  1. python3 -m venv fails on images whose Debian python has no ensurepip (The virtual environment was not created successfully because ensurepip is not available). python3 -m venv --system-site-packages --without-pip and then venv/bin/python -m pip install … works, because the image's pip is importable through --system-site-packages and installs into the venv prefix.
  2. pip install freetoken[accel] from git needs --no-build-isolation with setuptools>=77 wheel ninja installed first — an isolated build resolves a second torch and the C++ extension links against the wrong one.

With those two, the pytorch devel image works as a second base: driver ≥ 580 / nvcc 13.0 / torch 2.11.0+cu130 preflight, then ft serve — RTX 3090, Qwen3.8-Flash-Next NVFP4, coherent to 192k tokens (numbers on #293). One more thing the README could say: ft serve has no --api-key (#152), so a container that publishes 1919 is open — we run a small bearer-check proxy in front of it in the meantime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants