FlashInfer: Kernel Library for LLM Serving
-
Updated
Sep 11, 2026 - Cuda
#
distributed-inference
Here are 125 public repositories matching this topic...
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Sep 10, 2026 - Python
Achieve state of the art inference performance with modern accelerators on Kubernetes
-
Updated
Sep 11, 2026 - Shell
SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
text-to-speech streaming cuda inference pytorch tts speech-recognition music-generation asr model-serving multimodal openai-api audio-generation sglang distributed-inference
-
Updated
Sep 11, 2026 - Python
Best practices, reference architectures, and examples for distributed AI training and inference on AWS.
kubernetes aws gpu slurm efa distributed-training eks parallelcluster hyperpod generative-ai distributed-inference physical-ai
-
Updated
Sep 9, 2026 - Shell
Every device brings a slice. Together they run the whole model. Peer-to-peer LLM inference across browser tabs: a from-scratch WebGPU engine and a WebRTC runtime that split a 27B model over the devices in a room.
browser webrtc peer-to-peer inference webgpu wgsl llm local-llm speculative-decoding distributed-inference
-
Updated
Sep 9, 2026 - JavaScript
Split FLUX.2 and LTX 2.3 across two GPUs (LAN or same-machine) — NVENC compresses activations live on the wire. Icarus (ComfyUI node) + Daedalus (back-half server).
flux pytorch image-generation multi-gpu nvenc video-generation pipeline-parallelism diffusion-models flux2 comfyui distributed-inference comfyui-node ltx-video
-
Updated
May 17, 2026 - Python
[Official] Prima.cpp: Scale Your Local AI Beyond One Device.
-
Updated
Sep 6, 2026 - C++
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
-
Updated
Aug 23, 2026 - JavaScript
An educational distributed training and inference library for neural nets using local computing
-
Updated
Jun 10, 2026 - Python
Mixed-vendor GPU inference cluster manager with speculative decoding
python machine-learning deep-learning metal cuda p2p homelab rocm gpu-cluster llama-cpp gpu-inference local-llm llm-inference ollama speculative-decoding distributed-inference gpu-pooling self-hosted-ai openai-compatible mixed-gpu
-
Updated
Jul 2, 2026 - Python
Run any model on Intel silicon
-
Updated
Sep 11, 2026 - Rust
Decentralized peer-to-peer LLM inference network. Single Rust binary, BitTorrent-inspired incentives, OpenAI-compatible API.
rust mcp decentralized p2p inference transformer candle libp2p claude huggingface kv-cache openai-api llm gguf speculative-decoding anthropic-api distributed-inference claude-code openclaw openclaw-plugin
-
Updated
Sep 10, 2026 - Rust
Code for paper "JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks"
compression deep-learning distributed-computing inference dnn neural-networks pruning quantization dnn-optimization edge-computing mobilenet edge-cloud ai-models dnn-partition distributed-inference
-
Updated
Jul 22, 2026 - Python
Turn any Mac or GPU into an OpenAI-compatible inference node. One-command setup, automatic HTTPS, model management, and distributed request routing.
machine-learning gpu inference nvidia cloudflare llama mistral apple-silicon llama-cpp local-llm qwen gguf deepseek distributed-inference self-hosted-ai openai-compatible
-
Updated
Sep 5, 2026 - Python
An imperative command-line-interface for AI workload orchestration
kubernetes ray multi-cloud gpu-cluster mlops cloud-gpu mixture-of-experts huggingface vllm llm-inference ollama sglang agentic-workflow agentic-ai distributed-inference mcp-server gpu-provisioning disaggregated-inference
-
Updated
Sep 10, 2026 - Python
🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit-for-bit identical to single-machine. No torch.distributed, no datacenter.
metal peer-to-peer decentralized cuda transformers inference pytorch mps gemma model-parallelism heterogeneous-computing edge-ai pipeline-parallelism on-device-ai apple-silicon large-language-models llm llm-inference qwen distributed-inference
-
Updated
Jul 6, 2026 - Python
MiniMax-H3 multi-GPU parallel inference for ComfyUI | 多卡并行加速节点:2-8 GPU Ulysses sequence parallel, bit-identical video+audio generation
h3 ulysses minimax multi-gpu multigpu video-generation image-to-video video-audio text-to-video multi-gpu-inference comfyui sequence-parallelism distributed-inference comfyui-custom-node minimax-h3
-
Updated
Aug 14, 2026 - Python
Optimize and run consistent AI Functions.
s3 prompt data-engineering data-pipelines observability evaluation-framework synthetic-data unstructured-data mlops batch-inference prompt-engineering llm-inference evals distributed-inference evals-engineering
-
Updated
Aug 13, 2026 - Python
Add this topic to your repo
To associate your repository with the distributed-inference topic, visit your repo's landing page and select "manage topics."