Streaming: parakeet_capi_stream_feed leaks ~20-40 MB per second of audio; stream_free + stream_begin does not reclaim

[contractorwolf]

Summary

parakeet_capi_stream_feed grows RSS without bound — roughly 19–35 MB per second of audio fed — and neither parakeet_capi_stream_free + stream_begin nor anything else short of killing the process reclaims it. On an 8 GB unified-memory board this exhausts the pool in a couple of minutes of continuous streaming.

Reproduced with silence, so it is not the transcript or token history accumulating.

Environment

parakeet.cpp	v0.5.0 (1bfbebf)
Board	NVIDIA Jetson Orin Nano 8 GB (sm_87, unified memory)
OS	Ubuntu 22.04.5 aarch64, L4T R36.4.7 / JetPack 6.2.1, kernel 5.15.148-tegra
CUDA	12.6.68
Build	-DPARAKEET_GGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87 -DPARAKEET_SHARED=ON
Model	mudler/parakeet-cpp-gguf → realtime_eou_120m-v1-q8_0.gguf

Reproduction

// leak_repro.c — feeds 100 ms blocks of silence, frees every returned string
#include "parakeet_capi.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static long rss_mb(void) {
    FILE *f = fopen("/proc/self/status", "r");
    char line[256]; long kb = 0;
    while (fgets(line, sizeof line, f))
        if (!strncmp(line, "VmRSS:", 6)) { sscanf(line + 6, "%ld", &kb); break; }
    fclose(f); return kb / 1024;
}

int main(int argc, char **argv) {
    parakeet_ctx *ctx = parakeet_capi_load(argv[1]);
    parakeet_stream *s = parakeet_capi_stream_begin(ctx);
    const int N = 1600;                       // 100 ms at 16 kHz
    float *pcm = calloc(N, sizeof(float));    // silence
    long base = rss_mb();
    printf("after load:            RSS %4ld MB\n", base);
    for (int i = 1; i <= 300; i++) {          // 30 s of audio
        int eou = 0;
        char *t = parakeet_capi_stream_feed(s, pcm, N, &eou);
        if (t) parakeet_capi_free_string(t);
        if (i % 50 == 0)
            printf("fed %2d s of audio:     RSS %4ld MB  (+%ld)\n",
                   i / 10, rss_mb(), rss_mb() - base);
    }
    parakeet_capi_stream_free(s);
    s = parakeet_capi_stream_begin(ctx);
    printf("after free + begin:    RSS %4ld MB  (+%ld)\n", rss_mb(), rss_mb() - base);
    return 0;
}

cc leak_repro.c -I<src>/include -L<build> -lparakeet -o leak_repro
LD_LIBRARY_PATH=<build>:<build>/third_party/ggml/src:<build>/third_party/ggml/src/ggml-cuda \
  ./leak_repro realtime_eou_120m-v1-q8_0.gguf

Result

after load:            RSS  373 MB
fed  5 s of audio:     RSS  677 MB  (+304)
fed 10 s of audio:     RSS  767 MB  (+394)
fed 15 s of audio:     RSS  861 MB  (+488)
fed 20 s of audio:     RSS  946 MB  (+573)
fed 25 s of audio:     RSS 1042 MB  (+669)
fed 30 s of audio:     RSS 1141 MB  (+768)
after free + begin:    RSS 1141 MB  (+768)

Steady state after the initial allocation is about 19 MB per second of audio, linear, no plateau.

Independently via the Python/ctypes binding with real speech (a 9.4 s clip looped), the rate is higher — 28–41 MB per audio-second:

after load: 401 MB
 18.7s of speech -> 1178 MB  (41.4 MB per audio-second)
 37.5s of speech -> 1819 MB  (37.8)
 56.2s of speech -> 2469 MB  (36.8)
 75.0s of speech -> 2772 MB  (31.6)
 93.7s of speech -> 3038 MB  (28.1)

What does not help

• stream_free + stream_begin on the same ctx. The C repro above shows RSS unchanged across it. Cycling free/begin every 8 s of audio in Python still grew ~220 MB per cycle, so the retained memory appears to sit in the ctx or an allocator rather than the StreamingSession.
• Freeing every string returned by stream_feed (done in the repro).
• Draining events — the repro never enqueues any, and it leaks anyway.

Only killing the process reclaims it.

Why it matters here

This makes a long-running streaming session impossible on a memory-constrained board. It is compounded by ggml's behaviour on allocation failure: a failed CUDA allocation calls abort() rather than returning an error, so the process does not degrade, it core-dumps —

alloc_tensor_range: failed to allocate CUDA0 buffer of size 175952000
[parakeet] realize_weights: alloc_ctx_tensors failed
ggml-backend.cpp:342: GGML_ASSERT(buf != NULL && "tensor buffer not set") failed

My workaround is to run the streaming session in a child process and respawn it at every utterance boundary, which bounds the damage but costs a model reload per turn.

Also observed: core dump on clean exit

Separate from the leak, and visible at the end of the same repro run — normal process teardown aborts:

CUDA error: driver shutting down
  current device: -1, in function ~ggml_backend_cuda_buffer_context at ggml-cuda.cu:635
  cudaFree(dev_ptr)
ggml-cuda.cu:102: CUDA error
#4  ggml_backend_cuda_buffer_free_buffer(ggml_backend_buffer*)
#5  ggml_backend_buffer_free
#6  ggml_gallocr_free
#9  __cxa_finalize
#10 __do_global_dtors_aux () from libparakeet.so

A static/global destructor in libparakeet.so frees CUDA buffers after the driver has begun shutting down. Every clean exit leaves a core dump. Happy to file that separately if you would prefer it split out.

Not tested

Whether the leak also occurs on the CPU backend — I only have a CUDA shared build here. Glad to test anything that would help narrow it down.