Running large language models on a single GPU for throughput-oriented scenarios.
-
Updated
Oct 28, 2024 - Python
Running large language models on a single GPU for throughput-oriented scenarios.
Run Mixtral-8x7B models in Colab or consumer desktops
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
A QoE-Oriented Computation Offloading Algorithm based on Deep Reinforcement Learning (DRL) for Mobile Edge Computing (MEC) | This algorithm captures the dynamics of the MEC environment by integrating the Dueling Double Deep Q-Network (D3QN) model with Long Short-Term Memory (LSTM) networks.
ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory [COLM2025]
LLM Inference on consumer devices
dpdk infrastructure for software acceleration. Currently working on RX and ACL pre-filter
DPU-Powered File System Virtualization over virtio-fs
A collection of tests for the Open vSwitch HW offload.
Run 50-280 GB MoE LLMs (DeepSeek V4/V4.1 Flash, GLM-5.3-Flash, Qwen3.8-Flash-Next, Ornith) on one consumer GPU: experts spread over VRAM, RAM and NVMe SSD. Browser chat + OpenAI/Anthropic APIs. CUDA, ROCm, DGX Spark.
A Dynamic Programming Offloading Algorithm for Mobile Cloud Computing
Monero hardware wallet protocol implementation for Trezor, agent
LeapIO: Efficient and Portable Virtual NVMe Storage on ARM SoCs (ASPLOS'20)
A framework for IoT devices to offload tasks to the cloud, resulting in efficient computation and decreased cloud costs.
A lightweight framework that enables serverless users to reduce their bills by harvesting non-serverless compute resources such as their VMs, on-premise servers, or personal computers.
30 tok/s for 20B MoE on 8 GB VRAM. Flat throughput to 32K context. Native MXFP4 + GGUF Q4_K/Q5_K/Q6_K via ggml CUDA kernels — zero dequant. Expert offloading for models that don't fit in GPU memory.
The container-based cloud platform for mobile code offloading
Code for paper "Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI" (MobiCom'22)
Monero wallet Trezor integration documentation
To associate your repository with the offloading topic, visit your repo's landing page and select "manage topics."