🛠️[v0.4.2] Release Note: RPC Device Selection and Extended Batch / Runtime Bindings

[JamePeng]

[0.4.2] — RPC Device Selection and Extended Batch / Runtime Bindings

This release introduces a major expansion of RPC backend support, allowing Llama instances to explicitly select remote and local devices for model execution. RPC endpoints are now validated before registration, process-wide registrations are reused safely, and server-side configuration with prebuilt ggml-rpc-server from the official llama.cpp releases can place model layers entirely on selected remote devices when desired.

The release also expands the low-level runtime surface with the extended batch API, multi-tensor GGML buffer allocation bindings, and several ABI corrections across state, MTMD, quantization, optimizer, and callback interfaces. RPC is now enabled across CUDA and Metal wheel builds and verified in Linux, Windows, and macOS CI.

Highlights

• RPC device selection

  • Probe and validate RPC endpoints before native registration.
  • Reuse process-wide RPC registrations.
  • Select remote and local devices explicitly per Llama instance.
  • Validate device ordering and tensor splits.
  • Support remote-only model placement through rpc_local_devices=[].

• Server RPC support

  • Expose rpc_local_devices through server model settings.
  • Pass explicit device selection into model creation.

• Extended llama.cpp bindings

  • Add llama_batch_ext, llama_embd, llama_process_type, and llama_process.
  • Cover both token and embedding input paths.
  • Correct session and sequence-state return types.

• GGML backend expansion

  • Bind multi-tensor buffer allocation and allocation-size APIs.

• Build and CI

  • Enable RPC in CUDA and Metal wheels.
  • Enable RPC validation across Linux, Windows, and macOS tests.
  • Clean up wheel build configuration and update GitHub Actions dependencies.

• ABI fixes

  • Align MTMD post-decode callbacks with the embedding batch API.
  • Correct quantization, optimizer, and GGML optimizer callback structures and return types.

• Documentation

  • Add RPC setup, remote-device selection, troubleshooting, and multimodal offloading guides.
  • Include Qwen3.5 image examples using an RPC language model with an independently initialized MTMD projector.
  • Present the project Wiki as the actively maintained documentation source.

• API sync

  • Sync llama.cpp llama / MTMD / GGML API bindings to 2026-10-03.

Thanks to @alcoftTAO for RPC backend testing feedback in issue #181.

More information:
6332d8d...6126add

--JamePeng

[JamePeng]

Due to an increasing number of non-standard low-level development practices—particularly the excessive use of AI without regard for cross-platform compatibility—it has become difficult to implement certain C++ functionalities using ctypes (such as utilizing container classes like vector and map instead of passing data via C-style struct array pointers and element counts).

After repeated feedback yielded no results, I am releasing version 0.4.2 as a milestone to be archived via a tag;

Moving forward, I will develop a bridging library to handle data type conversions, thereby enabling support for the latest low-level features.