Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1617,6 +1617,76 @@ let response = try await session.respond(
)
```

llama.cpp's Metal backend stops a process with an assertion
if the process exits while a model has layers on the GPU,
and a generation that is still running at exit can crash on the GPU or the CPU.
So `LlamaLanguageModel` registers an exit handler when a load ends.
The handler stops new loads and generations,
waits up to three seconds for running work to stop at its next token,
and then frees every loaded model, projector, and cached context.
After the handler starts,
requests throw `LanguageModelSession.GenerationError.assetsUnavailable`.

The handler is best effort.
The process can still stop with the assertion in these cases:

- It exits during the first load,
because the handler is registered when a load ends.
- It exits from inside a load, a generation, or the release of a model,
for example from a llama.cpp callback.
That work can't end while the handler waits,
so the handler returns at once and frees nothing.
An exit from a llama.cpp callback on a thread
that llama.cpp or another library owns
isn't recognized as inside that work,
so the handler waits the full three seconds and then frees nothing.
- Work is still running after three seconds.
The handler then frees nothing.
- It exits without running exit handlers,
for example with `_exit` or after a crash.

If freeing the models takes more than ten seconds,
for example because another library's log callback blocks,
the handler aborts the process, so that the exit can't hang.
A crash report for this abort shows
`LlamaLifecycle.abortAfterExitCleanupTimeout()` in its backtrace,
which first tries to write one line to standard error,
but aborts whether or not the line can be written.
If the cleanup ends just after the limit,
the cleanup's own thread can abort first,
without the line,
and the backtrace shows `LlamaLifecycle.abortAfterLateExitCleanup()`.

No method unloads a model that something still references.
To be certain that no model is loaded when the process exits,
wait until every `respond` call returns and every response stream ends,
start no new requests,
and then release every reference to the `LlamaLanguageModel`,
including the sessions that use it.
The model frees its weights when the last reference is released.
A response stream's generation can keep its model for a short time
after the stream ends,
until the generation stops at its next token.
A `prewarm` keeps its model until its download and load end,
and you can't wait for it directly.
A request that you await after a `prewarm` waits for that load,
but the `prewarm` can keep the model for a short time after the request returns.

To be certain that the assertion can't happen,
set the environment variable `GGML_METAL_NO_RESIDENCY` to any value
before the first model loads.
llama.cpp's Metal backend then doesn't use residency sets,
so a process exits normally even with a model loaded on the GPU.
Set it in the process's environment,
or call `setenv("GGML_METAL_NO_RESIDENCY", "1", 1)` before the first request or `prewarm`,
because `LlamaLanguageModel` starts the llama.cpp backend at the first load.
If another library in the process uses llama.cpp,
set it before that library starts llama.cpp with `llama_backend_init`.
The cost is that residency sets keep a model's GPU memory wired between requests;
without them, the system can reclaim that memory while the GPU is idle,
and the next request waits for it again.
This package doesn't set the variable.

<a id="litert-lm-checkout-fails-with-a-git-lfs-smudge-error"></a>

### LiteRT-LM
Expand Down
161 changes: 133 additions & 28 deletions Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@ import Foundation
text: UnsafePointer<CChar>?,
userData: UnsafeMutableRawPointer?
) {
#if DEBUG
LlamaTestHooks.onLog.withLock { $0 }?()
#endif
let minimum = LlamaRuntime.logLevel.withLock { $0 }
let logger = LlamaRuntime.Logger(userData: userData)
guard LlamaRuntime.shouldWrite(level: level, logger: logger, minimum: minimum), let text else {
Expand All @@ -120,6 +123,57 @@ import Foundation
/// let model = LlamaLanguageModel(modelPath: "/path/to/model.gguf")
/// let hubModel = LlamaLanguageModel(modelId: "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M")
/// ```
///
/// llama.cpp's Metal backend stops a process with an assertion
/// if the process exits while a model has layers on the GPU.
/// So the model registers an exit handler when a load ends,
/// which stops new loads and generations,
/// waits up to three seconds for running work,
/// and then frees every loaded model.
/// After the handler starts, requests throw
/// ``LanguageModelSession/GenerationError/assetsUnavailable(_:)``.
/// The handler can't help in these cases:
///
/// - An exit during the first load.
/// - An exit from inside a load, a generation, or the release of a model,
/// such as from a llama.cpp callback.
/// An exit from a llama.cpp callback on a thread
/// that runs no task and releases no model,
/// such as a thread that llama.cpp or another library owns,
/// makes the handler wait the full three seconds and then free nothing.
/// - Work that is still running after three seconds.
/// - An exit that runs no exit handlers, such as `_exit` or a crash.
///
/// If freeing the models takes more than ten seconds,
/// the handler aborts the process, so that the exit can't hang.
/// A crash report shows `abortAfterExitCleanupTimeout()` in its backtrace,
/// which first tries to write one line to standard error.
/// If the cleanup ends just after the limit,
/// the cleanup's thread can abort first, without the line,
/// and the backtrace shows `abortAfterLateExitCleanup()`.
///
/// No method unloads a model that something still references.
/// To be certain that no model is loaded when the process exits,
/// wait until every request returns and every response stream ends,
/// start no new requests,
/// and then release every reference to the model, including its sessions.
/// A response stream's generation can keep the model for a short time
/// after the stream ends, until the generation stops at its next token.
/// A `prewarm` keeps the model until its download and load end,
/// and you can't wait for it directly.
/// A request that you await after a `prewarm` waits for that load,
/// but the `prewarm` can keep the model for a short time after the request returns.
///
/// To be certain that the assertion can't happen,
/// set the environment variable `GGML_METAL_NO_RESIDENCY` to any value
/// before the first model loads,
/// in the process's environment or with `setenv` before the first request or `prewarm`,
/// because the model starts the llama.cpp backend at the first load.
/// llama.cpp's Metal backend then doesn't use residency sets,
/// so a process exits normally even with a model loaded on the GPU.
/// The cost is that the system can reclaim a model's GPU memory
/// while the GPU is idle, and the next request waits for it again.
/// This package doesn't set the variable.
public final class LlamaLanguageModel: LanguageModel, @unchecked Sendable {
/// The reason the model is unavailable.
/// This model is always available.
Expand Down Expand Up @@ -565,7 +619,13 @@ import Foundation
/// frees a context that is still decoding: it runs on a transient context
/// instead and leaves the cache untouched.
private let sessionContextLock = NSLock()
private var cachedSessionContext: CachedSessionContext?
private var cachedSessionContext: CachedSessionContext? {
// The exit handler frees the cached context through the resources.
didSet { resources?.setCachedContext(cachedSessionContext?.context) }
}

/// The loaded model's resources, which the exit handler can free.
private var resources: LlamaModelResources?

/// The number of prompt tokens reused from the cached context by the most
/// recent chat generation.
Expand All @@ -580,21 +640,30 @@ import Foundation
/// model. The next chat generation prefills its full prompt again. Call this
/// under memory pressure or when a session is discarded. If a generation is
/// running on the cached context, it is freed as soon as that generation ends.
///
/// After the exit handler starts, this method does nothing,
/// because the handler frees the cached context.
public func clearCachedContext() {
discardCachedSessionContext()
// After the exit handler starts, the context may be freed already.
try? LlamaLifecycle.shared.perform { discardCachedSessionContext() }
}

private func discardCachedSessionContext() {
sessionContextLock.lock()
defer { sessionContextLock.unlock() }
guard var cached = cachedSessionContext else { return }
if cached.isCheckedOut {
cached.discardWhenReleased = true
cachedSessionContext = cached
return
let unused: OpaquePointer? = sessionContextLock.withLock {
guard var cached = cachedSessionContext else { return nil }
if cached.isCheckedOut {
cached.discardWhenReleased = true
cachedSessionContext = cached
return nil
}
cachedSessionContext = nil
return cached.context
}
// Free without the lock: llama.cpp logs while it frees a context,
// and a log callback that calls `exit` runs the exit handler on this thread.
if let unused {
llama_free(unused)
}
llama_free(cached.context)
cachedSessionContext = nil
}

private func recordCachedTokens(_ tokens: [llama_token], context: OpaquePointer) {
Expand All @@ -615,19 +684,22 @@ import Foundation
/// context is checked back in (or freed, if a discard was requested while
/// it was busy); a transient context is freed.
private func releaseSessionContext(_ context: OpaquePointer) {
sessionContextLock.lock()
defer { sessionContextLock.unlock() }
if var cached = cachedSessionContext, cached.context == context {
if cached.discardWhenReleased {
llama_free(cached.context)
cachedSessionContext = nil
} else {
let unused: OpaquePointer? = sessionContextLock.withLock {
if var cached = cachedSessionContext, cached.context == context {
if cached.discardWhenReleased {
cachedSessionContext = nil
return context
}
cached.isCheckedOut = false
cachedSessionContext = cached
return nil
}
return
return context
}
// Free without the lock, as in `discardCachedSessionContext()`.
if let unused {
llama_free(unused)
}
llama_free(context)
}

/// Returns a context for the session along with the index of the first
Expand Down Expand Up @@ -818,14 +890,10 @@ import Foundation
}

deinit {
if let cached = cachedSessionContext {
llama_free(cached.context)
}
if let mtmdContext = mtmdContext {
mtmd_free(mtmdContext)
}
if let model = model {
llama_model_free(model)
// The resources free the cached context, the projector, and the model,
// unless the exit handler freed them already.
if let resources {
LlamaLifecycle.shared.free(resources)
}
}

Expand Down Expand Up @@ -968,6 +1036,11 @@ import Foundation
let runtimeOptions = resolvedOptions(from: options)
let structuredOptions = resolvedStructuredOptions(from: options)

// The exit handler waits for this request's llama.cpp work,
// but not for its tool calls.
let operation = try LlamaLifecycle.shared.begin()
defer { operation.end() }

// Structured output does not go through the projector yet.
// Without an image marker, a prompt with an image throws
// before any generation or tool call,
Expand Down Expand Up @@ -1071,6 +1144,7 @@ import Foundation

try session.recordToolRound(toolRounds, completed: allEntries)

operation.suspend()
let (callsEntry, resolution) = try await resolveToolCalls(
parsedCalls,
requestContext: requestContext,
Expand Down Expand Up @@ -1104,7 +1178,9 @@ import Foundation
pendingEntries.append(.prompt(prompt))
}
}
try operation.resume()
}
try operation.resume()
if type == String.self {
return LanguageModelSession.Response(
content: text as! Content,
Expand Down Expand Up @@ -1215,6 +1291,10 @@ import Foundation

let runtimeOptions = resolvedOptions(from: options)
let maxTokens = runtimeOptions.maximumResponseTokens ?? 100
// The exit handler waits for this request's llama.cpp work,
// but not for its tool calls.
let operation = try LlamaLifecycle.shared.begin()
defer { operation.end() }
let outputFormat = self.currentToolCallFormat()
let toolRounds = session.toolRoundLimit(for: "Llama", options: options)
var accumulatedEntries: [Transcript.Entry] = []
Expand Down Expand Up @@ -1327,6 +1407,7 @@ import Foundation

try session.recordToolRound(toolRounds, completed: accumulatedEntries)

operation.suspend()
let (callsEntry, resolution) = try await self.resolveToolCalls(
parsedCalls,
requestContext: requestContext,
Expand Down Expand Up @@ -1359,6 +1440,7 @@ import Foundation
emittedBase += roundVisible
yieldSnapshot(emittedBase)
}
try operation.resume()
}

if emittedBase != lastYieldedText {
Expand All @@ -1382,6 +1464,8 @@ import Foundation

/// Downloads a Hub model's files if needed, then loads the model.
func ensureModelLoaded() async throws {
// After the exit handler starts, don't list or download a Hub model's files.
try LlamaLifecycle.shared.stopIfExiting()
let files =
if let hubSource {
try await hubSource.files()
Expand All @@ -1399,6 +1483,11 @@ import Foundation

guard !isModelLoaded else { return }

// The exit handler waits for this load.
// After the handler starts, this throws before the path is checked.
let operation = try LlamaLifecycle.shared.begin()
defer { operation.end() }

// Check if model file exists
guard FileManager.default.fileExists(atPath: modelPath) else {
throw LlamaLanguageModelError.invalidModelPath
Expand All @@ -1423,6 +1512,7 @@ import Foundation

let modelParams = createModelParams()
guard let loadedModel = llama_model_load_from_file(modelPath, modelParams) else {
try LlamaLifecycle.shared.stopIfExiting()
throw LlamaLanguageModelError.modelLoadFailed
}

Expand All @@ -1446,6 +1536,10 @@ import Foundation
self.vocab = llama_model_get_vocab(loadedModel)
self.tokenCache = StructuredGenerationTokenCache()
self.isModelLoaded = true
let resources = LlamaModelResources(model: loadedModel, projector: mtmdContext)
self.resources = resources
LlamaLifecycle.shared.track(resources)
LlamaRuntime.registerExitHandler()
}

private func createModelParams() -> llama_model_params {
Expand All @@ -1454,6 +1548,8 @@ import Foundation

// Try to reduce memory usage
params.load_mode = LLAMA_LOAD_MODE_MMAP
// Returning false stops the load, so that the exit handler doesn't wait for all of it.
params.progress_callback = { _, _ in !LlamaLifecycle.shared.isStopping }
return params
}

Expand Down Expand Up @@ -1737,6 +1833,7 @@ import Foundation
}

mutating func decode(_ token: Int) async throws {
try LlamaLifecycle.shared.stopIfExiting()
let llamaToken = llama_token(token)

batch.pointee.n_tokens = 1
Expand Down Expand Up @@ -1871,9 +1968,14 @@ import Foundation
var decodedTokens = promptTokens

for _ in 0 ..< maxTokens {
#if DEBUG
LlamaTestHooks.beforeToken.withLock { $0 }?()
#endif
if Task.isCancelled {
break
}
// The exit handler waits for this generation, so stop it early.
try LlamaLifecycle.shared.stopIfExiting()

// Sample next token from logits of the last token we just decoded
let nextToken = llama_sampler_sample(sampler, context, batch.n_tokens - 1)
Expand Down Expand Up @@ -2044,6 +2146,8 @@ import Foundation
if Task.isCancelled {
break
}
// The exit handler waits for this generation, so stop it early.
try LlamaLifecycle.shared.stopIfExiting()

let nextToken = llama_sampler_sample(samplerPtr, context, sampleIndex)
llama_sampler_accept(samplerPtr, nextToken)
Expand Down Expand Up @@ -2187,6 +2291,7 @@ import Foundation
let capacity = Int(batchSize)
var start = startIndex
while start < promptTokens.count {
try LlamaLifecycle.shared.stopIfExiting()
let count = min(capacity, promptTokens.count - start)
batch.n_tokens = Int32(count)
for i in 0 ..< count {
Expand Down
Loading
Loading