LIM (Local Inference Manager) is a C++ terminal-based LLM controller built on llama.cpp. It provides a persistent, stateful session where the KV-cache is never discarded: each turn's tokens are simply appended, so new input costs only O(input tokens), not O(total history). LIM has native filesystem tools for reading, searching, editing, and writing files, plus web searching and PDF reading.
Cumulative decode time vs context length for Qwen3.6-35B-A3B-UD-Q4_K_XL on an NVIDIA RTX 5090 / Intel i9-12900K. At 222041 context tokens, the default Mode 0 (LIM) is 1.8% faster than Mode 2 (CACHED) and 73% faster than Mode 1 (CHATBOT). See Benchmarking for how each mode works.
Median decode speed vs context length for Qwen3.8-27B-Q6_K on an NVIDIA RTX 5090 / Intel i9-12900K, with MTP enabled (draft=4). Aggregated per 10k context tokens over 294 sessions (~20.8k speed samples). The median falls from ~70 t/s at low context to ~46 t/s at the full 262144 context. MTP improves decode speed by ~25% over non-speculative decoding at all context lengths.
Every mainstream chatbot (and most local LLM frontends using server APIs) follows the reprocess-every-turn model: each time you send a message, the entire conversation history is re-transmitted in full and re-tokenized from scratch. The per-turn decode cost grows linearly with context length, making long conversations progressively slower.
LIM avoids this by design: it runs locally as a single persistent process where the KV-cache is never discarded. Each turn simply continues from where the last one left off. This is the same approach used by llama-cli in interactive mode. LIM includes an integrated tool system, session save/restore, interactive undo, and LaTeX-aware browser output.
+-------------+ +---------------+ +---------------+
| Your TTY |<--->| lim (C++) |<--->| llama.cpp |
| (readline) | | controller | | inference |
+-------------+ +-------+-------+ +---------------+
|
+-------------+-------------+
| | |
+-----v-----+ +----v-----+ +-----v------+
| Filesystem| | Web | | Browser |
| Tools | | Search | | (FIFO) |
| | | & PDFs | | |
+-----------+ +----------+ +------------+
lim: The C++ binary. Handles the REPL, KV-cache management, tool dispatch, and signal handling.limServer.py: An optional Python WebSocket server that streams output to a browser via the FIFO in$LIM_CONFIG_DIR. It is a long-lived service: the first lim session that needs it starts it on demand, and it survives across lim sessions (exit or crash) so the browser connection persists.
- A GPU with CUDA support (NVIDIA recommended) and the CUDA toolkit installed.
For CPU-only builds, run
make GGML_CUDA=off. - Python 3 with
aiohttpfor the browser server:pip3 install aiohttp. - A GGUF model file (e.g., Qwen, Llama, Mistral).
- Optional: SearXNG for web search and Docling for PDF reading. LIM auto-starts them on demand; override the commands with
LIM_SEARXNG_CMD/LIM_DOCLING_CMD(see Web Search & PDF Setup below).
Note: llama.cpp is bundled as a git subrepo with four in-tree patches (kept as
*.patchfiles): recurrent state checkpointing for instant/undoon hybrid models, UTF-8 robustness, MTP support (LIM_MTPonqwen4exp), and Q8_0 KV flash-attention.
git clone https://github.com/statefullm/lim.git
cd lim
make
./limThe build will automatically detect your GPU (if any) and compile llama with the appropriate architecture flags. No manual setup is required.
make also builds the VS Code extension (requires Node.js and npm; it is installed by make install). To build only the binary at any time, use make lim.
All auto-detected values can be overridden via environment variables:
| Variable | Purpose | Example |
|---|---|---|
CUDA_ARCH_FLAGS |
Override GPU architecture detection | make CUDA_ARCH_FLAGS=120a |
LIM_LLAMA_BUILD_DIR |
Use pre-built llama from external directory | export LIM_LLAMA_BUILD_DIR=/path/to/llama/build |
LIM_LLAMA_CMAKE_FLAGS |
Extra cmake flags passed to the llama build | make LIM_LLAMA_CMAKE_FLAGS="-DGGML_AVX512=off" |
GGML_CUDA |
Force CUDA on/off | make GGML_CUDA=off for CPU-only |
GGML_HIPBLAS |
Enable ROCm/HIP build | make GGML_HIPBLAS=on |
To clean only the llama build artifacts without touching lim:
make llama-cleanAfter building, install the binary, config files, and VS Code extension. This must be run as $LIM_AI_USER (e.g., su - $LIM_AI_USER) so that files are installed into that user's home directory, which is where LIM will look for them at runtime:
su - $LIM_AI_USER
make installThis installs:
limbinary to~/bin/lim(ensure~/binis in your PATH via.bashrc)- LLM-side git guard
llm/gitto~/bin/git relimto~/bin/relim- System prompt to
~/.config/lim/prompt - Reincarnate instructions to
~/.config/lim/reincarnate - Search cache directory at
~/.config/lim/searchCache - Browser server files (
limServer.py,viewer.html,viewer.js,libs/) to~/.config/lim/ - VS Code extension (built, then installed into this user's VS Code via
code)
All LIM configuration files live under $LIM_CONFIG_DIR, which defaults
to ~/.config/lim/.
The dedicated LLM user is controlled by the LIM_AI_USER environment variable (defaults to ai). All references below use $LIM_AI_USER. The group name always matches the user name.
The minimum for a first session is step 1 (creating the user), step 3 (denying cron/at), and steps 5 and 6 (connecting, bashrc): LIM is fully usable inside the AI's home without any of the sharing setup. Steps 2 and 4 are only needed to point it at your own project directories.
LIM must run as a dedicated user. The binary enforces this at startup: it checks getuid() and refuses to run otherwise. This is a security measure: the LLM has filesystem read/write access and shell execution capabilities, so isolating it behind a dedicated user confines its direct access to that account.
Create the user and group if they don't exist:
export LIM_AI_USER=ai
sudo groupadd -f $LIM_AI_USER
sudo useradd -m -g $LIM_AI_USER -s /bin/bash $LIM_AI_USER
sudo passwd -l $LIM_AI_USERThe account is intentionally passwordless -- do not set a password (sudo passwd -S $LIM_AI_USER should show L).
Then add your personal user to the $LIM_AI_USER group so you can read and write files the LLM creates without needing sudo chown:
sudo usermod -aG $LIM_AI_USER $USERLog out and back in (or run newgrp $LIM_AI_USER) for the group change to take effect.
Git refuses to operate in a repository owned by a different user ("detected dubious ownership"). Shared repos need no safe.directory entry for your personal user: aishare leaves the checkout and its stored gitdir owned by you. The refusal is also a security layer -- it keeps a gitdir the LLM created or swapped inert -- so do not add a blanket entry (directory = /home/$LIM_AI_USER, *).
The LLM account needs its own entry so LIM can run git in your user-owned repos; in $LIM_AI_USER's ~/.gitconfig add (substituting your username for $USER):
[safe]
directory = ~/*
directory = /home/$USER/*The LLM must not be able to schedule delayed jobs (they could fire during an aishare run and race its ownership changes). Add the LLM user to the system deny lists:
echo $LIM_AI_USER | sudo tee /etc/cron.deny
echo $LIM_AI_USER | sudo tee /etc/at.denyYour project directories should be group-writable by the $LIM_AI_USER group so the LLM can read and write files. Because you added your personal user to the $LIM_AI_USER group (step 1), access is symmetric: you can read files the LLM creates, and the LLM can read your files. The recommended layout for /home/$LIM_AI_USER:
$ ls -ld /home/$LIM_AI_USER
drwxrwsr-x. 42 $USER $LIM_AI_USER 4096 Jan 1 00:00 /home/$LIM_AI_USER/The setgid bit (s) ensures new files inherit the $LIM_AI_USER group.
Copy the tools. Copy aishare (grants a directory tree to the LLM) and the git wrapper (protects your git runs in shared repos) from this repository to your personal /home/$USER/bin/; the wrapper must come ahead of the real git in your PATH (check with type git):
cp user/aishare /home/$USER/bin/
cp user/git /home/$USER/bin/The wrapper locates the real git automatically (or $LIM_GIT_BIN if set). In a shared repo it briefly protects the gitdir from the LLM for the duration of your command, runs your git in a Landlock sandbox (disable with LIM_GIT_SANDBOX=0), and re-shares the gitdir's state files afterwards, so neither side gets blocked.
Install the sudo helper. aishare and the wrapper need one passwordless sudo rule, added once to your sudoers (sudo visudo, or a drop-in in /etc/sudoers.d/). It allows only a small root-owned helper, aishare-helper (in the LIM repo), which re-validates every path itself and only the fixed chown identities (a bare chown ... /home/$LIM_AI_USER/* rule is not safe -- see SECURITY.md). Install it before the first aishare run (mode 0700, root-owned, so the LLM user cannot modify its checks):
sudo install -m 0700 aishare-helper /usr/local/bin/aishare-helperSubstitute your username for $USER:
$USER ALL=(root) NOPASSWD: /usr/local/bin/aishare-helper
Set ptrace_scope. While lim is running, aishare requires kernel.yama.ptrace_scope of 1-3 (at 0 it refuses to run, and lim warns at startup). Check with cat /proc/sys/kernel/yama/ptrace_scope; to set it persistently:
echo "kernel.yama.ptrace_scope = 1" | sudo tee /etc/sysctl.d/60-yama-ptrace.conf && sudo sysctl --systemShare a directory. Run aishare on any directory under /home/$LIM_AI_USER you want the AI to access -- or with no argument, on the current directory:
aishare /home/$LIM_AI_USER/projectIt sets ownership to $USER:$LIM_AI_USER (via sudo), grants group read/write and setgid, and clears the sticky bit. While running, it blocks the LLM's exec_shell (its other tools keep working) and suspends the LLM's exec_shell command trees -- in-flight and previously backgrounded -- resuming them when done, so the LLM cannot race the ownership changes. A lock file, /var/tmp/lim.aishare.lock, provides mutual exclusion between aishare runs; an existing lock aborts the run -- wait for the other run to finish, or rm /var/tmp/lim.aishare.lock if no aishare is running.
Re-run aishare in a directory whenever you add new files or directories to it -- a new git repo included. New entries you create inherit the $LIM_AI_USER group (via setgid) but usually not the group-write bit (your umask), so the AI can read them but not write until aishare re-grants group read/write. Also, aishare applies the git protections described in SECURITY.md.
aishare will not operate on target directories outside /home/$LIM_AI_USER.
LIM must be run as user $LIM_AI_USER. The simplest approach is to SSH into the host as that user, then launch lim directly. This keeps the workflow consistent whether you're using VS Code or a standalone terminal. SSH is key-based only (the account has no password): add your public key to /home/$LIM_AI_USER/.ssh/authorized_keys.
Add this to the ~/.bashrc of $LIM_AI_USER:
alias coder='lim ~/models/Qwen3.8-27B-Q6_K.gguf'Replace the model path with whichever GGUF you want to use. Then, before running coder, connect to the LIM server $LIM_HOST. The -L forward carries the browser viewer: the server listens on 127.0.0.1 only, so from your laptop the viewer is at the URL lim prints (see Browser Output -- the page needs a per-server-start connection token, so the bare viewer.html no longer connects):
ssh -a -L 8765:localhost:8765 $LIM_AI_USER@$LIM_HOST
coderIf running locally (no SSH needed), you have two options:
Option A: Switch users fully
sudo -iu $LIM_AI_USER
coderOption B: Launch from your personal shell via sudo
Add this function to your ~/.bashrc:
coder() { sudo -iu $LIM_AI_USER sh -c "cd \"$PWD\"; lim ~/models/Qwen3.8-27B-Q6_K.gguf $*"; }You can then start lim as user $LIM_AI_USER in your current directory:
coderLIM automatically detects P-cores and E-cores on hybrid CPUs (Intel Alder Lake, Raptor Lake, etc.) and pins background services accordingly:
| Workload | Default Cores | Description |
|---|---|---|
| SearxNG search server | E-cores | Light background Python process |
| Browser WebSocket server | E-cores | Light background Python process |
| Docling PDF converter | P-cores | Heavy ML inference workload |
On non-hybrid CPUs (all cores identical), no taskset pinning is applied. If the pinning command isn't found on $PATH (e.g., macOS has no taskset), pinning is silently skipped.
Override the auto-detected layout with LIM_TASKSET:
# Format: LIM_TASKSET="P_CORES:E_CORES"
export LIM_TASKSET="0-15:16-23" # Classic i9-12900K layout
export LIM_TASKSET="0-7:" # P-cores 0-7, no E-core pinning
export LIM_TASKSET=":8-15" # No P-core pinning, E-cores 8-15
export LIM_TASKSET="::" # Disable all taskset pinningmacOS / non-Linux systems: macOS has no taskset binary. To enable manual core pinning, install an alternative and point LIM_TASKSET_CMD at it:
# Install numactl via Homebrew, then use it as the pinning backend
brew install numactl
export LIM_TASKSET="0-3:4-7"
export LIM_TASKSET_CMD="numactl --physcpubind"Or write a custom wrapper script and set LIM_TASKSET_CMD to its path. If the command isn't on $PATH, pinning is silently skipped -- services still start normally, just unpinned.
The detected topology and pinning status are logged to stderr at startup when LIM_DEBUG=1.
Add these lines to /home/$LIM_AI_USER/.bashrc:
umask 0002 # Group-writable files by default
export PATH="$HOME/bin:$PATH"The system prompt lives at ~/.config/lim/prompt. A default prompt file ships with this repository and is installed to that location by make install. This file is read at startup and baked into the KV-cache (and re-read from disk on /clear and /reincarnate). It defines the LLM's behavior: available tools, editing workflow, formatting rules, etc. You can customize it for different use cases (coding assistant, writer, researcher, etc.). An optional directory-specific file localprompt (with a fallback to $LIM_CONFIG_DIR/localprompt) will be prepended to the system prompt.
Reasoning effort: For models that support reasoning-effort steering, add a reasoning-effort instruction to localprompt, matching how Qwen's own template places it. Suggested Qwen 3.8 instructions:
Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.
Omitting any such instruction leaves the model at its baseline behavior. For a full setup -- mode files, per-mode sampling env, and a login hook that keeps the two consistent -- see REASONING.md.
Create /home/$LIM_AI_USER/.lim_aliases to define shorthand expansions at the >>> prompt:
# ~/.lim_aliases: one key=value per line; # comments are ignored
# Keys must start with / to distinguish them from regular messages
/test=Test the filesystem tools and clean up any new files you created.
/message=Show me a 1-sentence concise git commit message relative to HEAD.
/commit=commit
/make=run make
/amend=Amend the previous commit.
/prompt=Follow the system prompt strictly!
/tools=Use our robust filesystem tools.
/pairs=Use search_file+edit_file pairs; ensure that OLD is a contiguous substring of the search_file result.
/rawtags=Raw function opening and closing tags are reserved for executing tool calls. They can appear in tool call content, but obviously you never want to output them in prose: use FUNC_START and FUNC_END instead.
/closing=Did you forget a closing parameter or function tag in your tool call?
At the >>> prompt, typing /test expands to Test the filesystem tools and clean up any new files you created. before being sent to the LLM. Lines starting with # are treated as comments. Blank lines are skipped. Built-in command names (e.g., /quit, /clear) cannot be used as alias keys. This is loaded fresh each session.
Set via LIM_OUTPUT:
| Value | stdout | Browser | Description |
|---|---|---|---|
| 3 | Y | Y | Both stdout and browser |
| 2 (default) | N | Y | Browser only |
| 1 | Y | N | Stdout only |
| 0 | N | N | No LLM output |
| Variable | Default | Description |
|---|---|---|
LIM_AI_USER |
ai |
Username that LIM must run as. Used by the binary for the user check, and by the VS Code extension for SSH. |
LIM_GIT_BIN |
(auto) | Path to the real git binary, used by both the user-side git wrapper and the LLM-side guard. Both auto-detect it as the first other git in PATH. |
LIM_GIT_SANDBOX |
1 |
git wrapper: run your git in a Landlock sandbox when a command reaches a shared repo (writes only under /home/$LIM_AI_USER, no new privileges). Set to 0 to disable; see User Setup and SECURITY.md. |
LIM_CACHE_DIR |
.cache |
Directory for fast restore KV-cache files. A relative value follows the working directory: /cd and /load re-anchor it to the new location, as with a relaunch. |
LIM_CONFIG_DIR |
~/.config/lim |
Directory for LIM config and server files (prompt, reincarnate, userprompt, searchCache, limServer.py, viewer.html, viewer.js, libs/) and runtime files (lim.fifo, lim.server_ready, lim.browser_ready, lim.server.pid, lim.viewer_url -- the tokenized viewer URL, mode 0640; see Browser Output). The C++ binary starts the browser server from here on first need (it then persists across sessions). Override to place everything elsewhere. |
LIM_LOG_DIR |
log |
Directory for session logs, token traces, and autosave files. A relative value follows the working directory: /cd and /load re-anchor it to the new location, as with a relaunch. |
LIM_SAVE_DIR |
. |
Directory prepended to relative paths in /save, /load, /delete, and the CLI restore argument. Absolute paths (starting with /) are unaffected. |
LIM_HOST |
unset | Hostname or IP of your LIM server. Used for the SSH connection; the viewer is always reached via localhost (through the SSH port forward when remote). |
LIM_PORT |
8765 |
Port for the browser WebSocket server (listens on 127.0.0.1 only; the SSH tunnel's remote side uses the same port) |
LIM_OUTPUT |
2 |
Output mode: 0 = none, 1 = stdout only, 2 = browser only (default), 3 = both |
LIM_VIEWER_URL |
(auto: http://localhost:<port>/viewer.html) |
Override the base viewer URL, e.g. when the SSH tunnel's local side uses a different port. The connection token (see Browser Output) is appended automatically, and the /ws Origin check accepts this URL's origin |
LIM_WEB_CONTEXT_FRACTION |
0.75 |
Fraction of LIM_CTX reserved for fetched web content. The per-file limit (LIM_WEB_FILE_MAX) defaults to 25% of this budget. Set between 0.0 and 1.0; 0.0 means the default 0.75. |
LIM_WEB_FILE_MAX |
(auto) | Max characters per fetched file before middle-drop truncation. Defaults to 25% of the session web budget. Set explicitly to override. |
LIM_WEB_HTML_MAX |
500000 |
Max bytes to buffer when downloading HTML/text pages via curl |
LIM_WEB_PDF_MAX |
50000000 |
Max bytes to buffer when downloading PDFs via curl (50 MB) |
LIM_WEB_TIMEOUT |
600 |
HTTP request timeout in seconds for fetches and searches |
LIM_BRAVE_API_KEY |
(empty) | Brave Search API key. When set, LIM falls back to the Brave Search API if SearXNG fails or returns no results. |
GH_TOKEN |
(empty) | Optional GitHub token for higher API rate limits when read_files fetches GitHub URLs (issues, PRs, discussions, files). Use a fine-grained token with read-only access to public repositories only -- the model can read and use this token. |
LIM_SEARCH_COOLDOWN |
3 |
Minimum seconds between web searches to avoid rate-limiting SearxNG |
LIM_DOCLING_CMD |
~/venv/bin/docling-serve run --enable-ui --host 127.0.0.1 |
Command to start the Docling PDF service. Binds to localhost only. Override if installed elsewhere (e.g., via Docker or a different venv). |
LIM_SEARXNG_CMD |
cd ~/searxng && exec python -m searx.webapp |
Command to start the SearxNG search service. Override if installed elsewhere. |
LIM_DEBUG |
0 |
Set to 1 for verbose token-level logging in $LIM_LOG_DIR/<N>.tokens |
LIM_LLAMA_DEBUG_LOG |
0 |
Only with LIM_DEBUG=1: set to 1 to also print DEBUG-level llama.cpp/ggml log lines (e.g. "CUDA Graph id N reused" on every decode). Default keeps INFO and above. |
LIM_EOG_RESAMPLE_MAX |
256 |
Maximum resampling attempts when the model emits a spurious EOG token before finishing its response. Increase if you see premature turn endings. |
LIM_TOOL_IGNORE |
100 |
Maximum tokens allowed outside parameters while inside a tool call before attempting a tool correction. |
LIM_GPU_LAYERS |
-1 |
Number of layers offloaded to GPU (-1 = auto-fit all layers). When set explicitly, bypasses auto-fitting. For MoE models that exceed VRAM, auto-fit uses partial layer offloading (dense weights on GPU, sparse expert weights on CPU) for optimal throughput. |
LIM_HONEST_SPEED |
0 |
Set to 1 for an "honest" wall-clock speed diagnostic: tokens / total generate() time, including all CPU-side overhead (sampling, output rendering, tool-call detection). Default 0 reports generation time matching llama-cli's "Generation: X t/s" (first-token to last-token sample+sync). |
LIM_SPEED_INTERVAL |
100 |
Number of tokens between in-loop speed diagnostic updates. |
LIM_USE_MLOCK |
1 |
Lock model in RAM to prevent swapping |
LIM_USE_MMAP |
0 |
Use memory-mapped model loading (faster startup, more RAM pressure) |
LIM_BATCH |
2048 |
Batch size for token feeding |
LIM_CTX |
262144 |
Context window size (KV-cache token capacity) |
LIM_MTP |
0 |
Set to 1 to enable MTP (multi-token-prediction) speculative decoding: a second context mirroring the main KV cache proposes up to LIM_MTP_DRAFT tokens per round, verified by the main model in a single batch decode. Speculation pauses when the mirror can't stay in sync (KV-exhaustion truncation, fast-restore from a cache saved without MTP) or after sustained low acceptance (auto-disables). Not implemented for benchmark chatbot modes 1 and 2. |
LIM_MTP_BATCH |
128 |
Cap on the MTP draft context's batch size (1-512, only with LIM_MTP=1). Lowering it saves VRAM at the cost of a slightly slower one-time prefill mirror. The actual draft-context memory is printed at startup with LIM_DEBUG=1. |
LIM_MTP_DRAFT |
4 |
Number of tokens proposed per MTP round (1-32, only with LIM_MTP=1); must not exceed the model's recurrent rollback window. The default was found to be the speed sweet spot for the Qwen3.8-27B class -- longer drafts cost more memory for diminishing acceptance gains. |
LIM_MTP_SIDECAR |
(empty) | Path to a separate MTP-only GGUF file (only with LIM_MTP=1). When set, the MTP draft head is loaded from this file instead of the main model. Use this for models whose GGUF quantization omits the nextn tensors (e.g. some Qwen3.8-Flash-Next builds). The sidecar is a small GGUF containing only the MTP layer weights (~2 GB) with nextn_predict_layers in its metadata. |
LIM_THREADS |
(auto) | Threads for inference: physical P-core count on hybrid CPUs (E-cores excluded), physical core count on non-hybrid CPUs, logical core count if the CPU topology can't be read |
LIM_THREADS_BATCH |
(auto) | Threads for batch processing (same default as LIM_THREADS) |
LIM_UBATCH |
512 |
Unbatched size |
LIM_MIN_P |
0.0 |
Minimum probability threshold: keep tokens where P >= min_p * P(top) |
LIM_FREQUENCY_PENALTY |
0.0 |
Frequency penalty: discourages overused tokens proportional to frequency |
LIM_PRESENCE_PENALTY |
1.5 |
Presence penalty: discourages repeating previously used tokens |
LIM_REPETITION_PENALTY |
1.0 |
Repetition penalty multiplier (1.0 = no penalty) |
LIM_SEED |
(auto) | Random seed for reproducibility (default: random) |
LIM_TEMPERATURE |
0.7 |
Sampling temperature (set to 0 for deterministic/greedy decoding). At 0 LIM also omits the wall-clock timestamp from the system prompt (the only session-varying prompt content), making runs byte-reproducible; the model does not know the current date/time in this mode -- pass it explicitly in the prompt if needed. |
LIM_THINKING |
1 |
Set to 0 to suppress thinking blocks via a pre-filled stub for faster throughput. Not recommended for math or complex reasoning tasks, as it can cause incorrect answers by skipping intermediate steps. |
LIM_DUMMY_THOUGHT |
(built-in) | Pre-filled stub used when LIM_THINKING=0. Leave unset to use the built-in default, set to an empty string to emit an empty thinking block (Qwen 3.8's "no thinking" signal), or set to any other string. |
LIM_ESCAPE_CONTRACT |
0 |
Set to 1 to include the reserved-token escape contract in the system prompt (the mechanism itself is always active; this only controls whether the LLM sees the explicit rules). Skipped when the base prompt file is empty. |
LIM_INLINE_LATEX |
1 |
Set to 0 to disable inline KaTeX rendering in the browser viewer. In default mode, inline $...$ candidates are validated before rendering: a math delimiter never abuts whitespace, backticks never appear in math, and the candidate must parse as KaTeX. Code spans / fenced blocks are never treated as math. Block-level $$...$$ math always renders. |
LIM_TOP_K |
20 |
Keep only the top_k most likely tokens (applied after the penalty sampler, before top_p / min_p / temperature) |
LIM_TOP_P |
0.8 |
Nucleus sampling: consider tokens with cumulative probability <= top_p |
LIM_CACHE_TYPE_K |
Q8_0 |
KV-cache key storage type (F16, Q4_0, Q5_0, Q5_1, Q8_0, Q8_1) |
LIM_CACHE_TYPE_V |
Q8_0 |
KV-cache value storage type (F16, Q4_0, Q5_0, Q5_1, Q8_0, Q8_1) |
LIM_CHATBOT_MODE |
0 |
Benchmarking mode: 0 = LIM normal (persistent KV-cache, same as llama-cli interactive), 1 = standard chatbot (re-tokenize + re-decode full history each turn), 2 = cache-aware prefix match (emulates llama-server). Modes 1 and 2 force honest speed measurement. |
LIM_EXEC_TRUNCATION |
32768 |
Maximum bytes of exec_shell output before truncation |
LIM_MAX_AUTO_CONTINUE |
500 |
Maximum depth of automatic tool-call chaining |
LIM_TURN_TIMEOUT |
3600 |
Maximum seconds per generation turn before auto-abort |
LIM_TASKSET |
(auto) | Format: "P_CORES:E_CORES" (e.g., "0-15:16-23"). Auto-detected on hybrid CPUs. Set to "::" to disable all pinning. |
LIM_TASKSET_CMD |
taskset -c |
Override the core-pinning command; on macOS (no taskset), brew install numactl and set to numactl --physcpubind. If the command isn't on $PATH, pinning is silently skipped. |
Between user turns, LIM resets the sampler chain (penalties ring buffer and RNG seed). This means repetition penalties apply only to tokens generated during the current turn, not to stale tokens from previous responses, and not to the user's input. During auto-continue (tool-call chains, /continue), the sampler state is preserved so generation continues seamlessly.
LIM runs as the dedicated $LIM_AI_USER account (default ai). Within that account, the LLM has full privileges: it can read, write, and execute anything the $LIM_AI_USER user can. The account is a regular user without sudo privileges. The isolation covers the LLM's direct access only; the code, commits, and commands it produces are for you to review before you run or push them.
The practical consequences:
- File write access is limited to directories where the
$LIM_AI_USERgroup has write permission (read access is broader -- the LLM can read anything the user can). Useaishareto grant write access to new project directories and files. Files created by the LLM are owned by$LIM_AI_USER:$LIM_AI_USERwith group read/write permissions (viaumask 0002), so your personal user can access them directly since it is a member of the$LIM_AI_USERgroup. - Shell commands executed via the
exec_shelltool run as user$LIM_AI_USER. They inherit that user's PATH and environment, and run in the directory where lim was launched (acdinside a command is local to that command). - Git integration: The AI can clone public repos and commit locally, but should not be given the authentication needed to push changes back to remote repositories. Shared repos' gitdirs are moved into a pinned, protected store (SECURITY.md) so a planted hook or a rewritten local config cannot execute as you on your next commit.
SECURITY.md documents the threat model and how each defense works -- the gitdir store, the git wrapper, the aishare suspension, and the sudo helper.
SSH in as $LIM_AI_USER, then run your coder alias. You'll see the >>> prompt. Type your request and press Enter.
The LLM has access to six tools. You don't call them directly; just describe what you want and the LLM handles the rest:
read_files: Read text files, PDFs, or URLssearch_file: Search for text within a file, with optional line rangeedit_file: Replace exact text within a file (surgical editing)write_file: Write or overwrite a fileexec_shell: Run a shell command as$LIM_AI_USERweb_search: Search the web via SearXNG
Tools are invoked via XML-structured tags in the LLM's output. Results are fed back as continuation tokens into the KV-cache, avoiding the overhead of re-tokenizing the full conversation history.
The readline interface binds Ctrl+J to insert a literal newline into your input, while Enter (Return) submits the line. This lets you compose multi-line messages:
>>> Here's a function I need fixed:
(ctrl-j)
(ctrl-j)
def foo(x):
(ctrl-j)
return x + 1
(ctrl-j)
(ctrl-j)
Please fix the type hints. <-- Enter submits
The binding is restored after each input cycle so it doesn't leak into your shell session.
The prompt uses GNU readline in callback mode with select() polling instead of blocking I/O. This allows Ctrl+C interrupts to be handled immediately during user input, not just during generation. All standard Emacs-style bindings are available:
| Keys | Action |
|---|---|
| Ctrl+A / Ctrl+E | Beginning / end of line |
| Ctrl+B / Ctrl+F | Backward / forward character |
| Alt+B / Alt+F | Backward / forward word |
| Ctrl+K / Ctrl+U | Kill to end / start of line |
| Ctrl+W | Kill word backward |
| Ctrl+Y | Yank (paste) |
| Ctrl+_ | Undo |
| Tab | Completion (if configured) |
| Up / Down | History navigation |
| Ctrl+R | Reverse search history |
| Command | Effect |
|---|---|
/clear |
Auto-save the current state to $LIM_LOG_DIR/<N>-clear.save, then clear the KV-cache (reset to system prompt only). The auto-saved file lets you restore if you change your mind. Use /save <name> to create a permanent restore point before clearing. |
/cd <dir> |
Change the working directory for the next session. Only allowed with an empty context (at startup or after /clear): re-anchors the session to the new directory (logs, cache, history) and re-feeds the system prompt there, so a new conversation can start there without quitting and reloading the model. |
/undo |
Interactive undo: auto-saves first to $LIM_LOG_DIR/<N>-clear.save, then presents an Undo> prompt listing all checkpoints (most recent first). Use up/down arrows to navigate, Enter to confirm. Ctrl+C, Ctrl+D, /quit, or /exit cancel the undo and return to the user prompt without losing your session. Selecting a checkpoint restores the session to the end of the turn associated with that prompt. On hybrid models, instant undo works only for checkpoints generated in the current session; pre-restore checkpoints from a fast cache restore require re-decode fallback. If the re-decode fallback is interrupted (Ctrl+C), LIM resets the context to a fresh session. |
/continue |
Resume a live interruption (only). If interrupted mid-tool-call, resumes from the exact point of interruption. At a well-formed boundary, keeps the model generating. After /undo to a mid-turn checkpoint it is a silent no-op -- type a prompt (e.g., "continue") to proceed. |
/reset |
Reset terminal, web search, and the browser server (if stopped). Useful for recovering from a corrupted terminal or disabled web search after an interrupt or connection failure. |
/reincarnate |
Ask the LLM to compose a new prompt in $LIM_CONFIG_DIR/userprompt, then clear and restart with it. |
/remind |
Re-send the system prompt to re-anchor the LLM. |
/save |
Save the full session state to $LIM_LOG_DIR/<N>.save, overwriting any previous save for this session. |
/save <path> |
Save the full session state to <path>.save. The path can be relative or absolute. If it already ends in .save, no extra extension is added. Use this to create named restore points at meaningful moments in your session. |
/quit or /exit |
Auto-save the current state to $LIM_LOG_DIR/<N>.save, then exit the session. |
/quit <path> or /exit <path> |
Named save with fast cache to <path>.save, then exit the session. Accepts the same path format as /save. |
/load <path> [--checkpoints] |
Load a saved session from within the current session. Must be used immediately after /clear: fails if any conversation tokens have been added since the clear. Accepts the same path format as /save: .save is appended automatically if not already present. Uses the fast cache when available, falling back to a re-decode that first offers checkpoint selection at a Restore> prompt. --checkpoints skips the fast cache (forcing the prompt). A full restore still auto-writes the fast cache; a partial (checkpoint-selected) restore does not. |
/delete <path> |
Delete a save file and any associated fast cache entry. Accepts the same path format as /save: .save is appended automatically if not already present. |
/help |
Display a summary of all available commands. |
You can save a running session and restore it later with zero context loss:
Save: Type /save at the >>> prompt to save to $LIM_LOG_DIR/<N>.save (overwrites any previous save for this session). Use /save <path> to create named checkpoints: e.g., /save cats saves to cats.save, and /save /tmp/checkpoint saves to /tmp/checkpoint.save. If the path already ends in .save, no extra extension is added. The save file contains only the conversation token sequence, keeping it small. Named saves also cache the full KV-cache for fast future restores.
Restore: Pass a save file as the last argument to coder (.save appended automatically if missing). The CLI restore runs the same code path as in-session /load, so behavior is identical in both, including --checkpoints:
coder $LIM_LOG_DIR/5.save # explicit extension
coder $LIM_LOG_DIR/5 # .save appended automatically
coder cats # restores from cats.saveThis restores the session exactly as it was: the full conversation, KV-cache position, and generation state. It also restores the working directory: save files record the directory they were made in, and a restore changes to it and re-anchors the session there (logs, cache, history) before decoding; if the directory no longer exists or is inaccessible, the restore is refused. Typing /clear after a restore resets to a fresh system prompt with the current date and working directory (after first auto-saving the restored state).
In-session load: Restore from within a running session with /load <path>, immediately after /clear (it fails if any conversation tokens were added since). It uses the same path conventions as /save (.save appended automatically) and the fast cache when available. Typical workflow:
>>> /clear
Context Cleared Successfully
>>> /load cats
Restoring session from cats.save... (75432 tokens)
Session #5 restored: 75432 tokens loaded (28%)
Partial restore via checkpoints: Save files record a checkpoint at the end of each conversation turn -- plus one at the 90% context point for turns that crossed it (see Automatic 90% Context Checkpoint) -- with your prompt text and the token position. Whenever a restore can't use the fast cache (or with --checkpoints), LIM offers checkpoint selection at a Restore> prompt before decoding: navigate with the arrow keys (most recent first), Enter to confirm (no match restores all tokens by default), or Ctrl+C / Ctrl+D / /quit to cancel and continue fresh. Restoring to a checkpoint replays tokens only up to the end of that turn, as if you had just received that response. Checkpoints accumulate across restore/save cycles.
Checkpoint restore: Add --checkpoints to skip the fast-format cache, triggering the checkpoint selection prompt even when a fast cache exists. This also rebuilds recurrent-state checkpoints at each prompt boundary during decode, enabling instant /undo for all historical turns on hybrid models (Qwen3.5/3.6). The flag works on the command line (after the save file) and with the in-session /load command, so there is no need to exit and restart the model:
coder cats --checkpoints# Or, within a running session:
>>> /clear
>>> /load cats --checkpoints
Fast restore cache: On first restore, tokens are decoded through the model to rebuild the KV-cache (regenerating recurrent-state checkpoints so /undo works instantly for hybrid models), and the rebuilt cache is written to $LIM_CACHE_DIR so all subsequent restores from the same save file are fast; with MTP enabled, the draft mirror KV is stored too, so a fast restore keeps MTP active. Named saves (e.g., /save cats) write the fast cache immediately; unnamed /save and auto-saves from /quit, /clear, and /reincarnate skip it to save space, relying on the cache built at first restore. Fast-cache restores build up recurrent checkpoints as new turns complete, supporting instant /undo for each new turn. $LIM_CACHE_DIR/ is safe to delete at any time; it is regenerated on the next restore.
Auto-save on clear and quit: Before clearing the context, LIM automatically saves the current state to $LIM_LOG_DIR/<N>-clear.save. Before exiting, it saves to $LIM_LOG_DIR/<N>.save. These use different filenames so neither clobbers the other: if you clear and then exit, both the pre-clear and post-clear states are preserved. To keep a permanent checkpoint at any point, use /save <name>.
Press Ctrl+C during generation to interrupt. The partial output is preserved in the KV-cache. Type /continue to resume seamlessly: the LLM doesn't even know it was interrupted. This works because the KV-cache still holds all generated tokens up to the interruption point.
If the model starts simulating the conversation inside its thinking block -- a second open think tag with tool calls and fabricated tool results as literal text -- LIM aborts generation a few tokens in and automatically feeds a fresh "continue" user turn. The aborted tokens stay in the context, retries are bounded by LIM_MAX_AUTO_CONTINUE, and the session ejects to the prompt when the budget is exhausted. The tripwire fires on any nested open think tag, so one emitted as deliberate content is recovered the same way.
LIM survives terminal disconnection. When your SSH connection drops, the pty sends SIGHUP to the session, which LIM ignores. The session, KV-cache, and any in-flight generation continue; the browser keeps streaming (default LIM_OUTPUT=2).
To reconnect, open a new terminal on the host and run relim. If lim does not respond as soon as reptyr takes the terminal, press Ctrl+C: it is forwarded to lim and wakes it, and the terminal reinitializes automatically on the first input. A detached lim auto-saves when stopped with kill.
Note: relim needs a patched reptyr -- vanilla 0.10.0 can hang silently when attaching a multithreaded target. reptyr-attach-retry.patch fixes the hang. Build it from the official source (tag reptyr-0.10.0) with the patch, and install the binary ahead of any packaged reptyr in PATH.
When a turn crosses 90% of the context window, LIM stores an extra mid-turn checkpoint labelled "/continue" before the real turn end.
By default (LIM_OUTPUT=2), LIM streams output to a browser via WebSocket. The server listens on 127.0.0.1 only, so the viewer is reachable from the machine itself or through an SSH tunnel.
The page requires a per-server-start connection token, so load the tokenized URL lim prints (also stored at $LIM_CONFIG_DIR/lim.viewer_url, mode 0640, readable only by you and $LIM_AI_USER):
http://localhost:8765/viewer.html#token=...
while the server is running -- from your local browser over the tunnel when remote (the port follows LIM_PORT; if the tunnel's local side uses a different port, set LIM_VIEWER_URL). The token rides in the URL's # fragment, which the browser never sends to the server; the /ws handshake must present it with a matching Origin. A bare viewer.html cannot subscribe, and other local users cannot read the 0640 URL file.
The token changes each time the server starts (reboot, /reset, or a replaced FIFO node) -- an old page or token no longer connects. Only on the first session after a reboot (no server yet) must you start lim before loading the URL; afterwards the server is already running, so the page can be loaded -- or left open -- before lim starts.
If lim starts with the viewer not connected, it prints the URL and waits before the first decode; the session proceeds as soon as the page loads. Ctrl+C gives up the wait: with LIM_OUTPUT=3 it proceeds with both outputs, otherwise browser output is disabled for the session and re-enables if the viewer connects later.
The Python server (limServer.py) is a persistent service: started on first need by the first lim session (pinned to efficiency cores on hybrid CPUs) and outliving lim sessions. It reads from a FIFO at $LIM_CONFIG_DIR/lim.fifo and broadcasts to all connected WebSocket clients, so the browser tab stays connected across lim restarts -- each new session (restart or /clear) streams a -- Session N -- divider above the previous session's output. The only reload is after a reboot or a closed tab. If the server stops -- or is left reading a deleted/replaced FIFO node -- a fresh server takes its place with a fresh token; load the URL its lim prints (or cat $LIM_CONFIG_DIR/lim.viewer_url).
The LIM Workspace extension provides a convenient way to set up your workspace: it creates an integrated terminal and opens a browser viewer once the server is ready.
The extension does two things:
- Opens a browser at the tokenized viewer URL (see Browser Output): it reads
lim.viewer_urlfrom the server's$LIM_CONFIG_DIR(its own environment, else theLIM_CONFIG_DIR=...line in$LIM_AI_USER's~/.bashrc, else the default~/.config/lim), directly from disk when local, over SSH when remote, after waiting for the server. If it cannot read the file, it opens the bare page, which tells you to load the URL lim printed. - Creates a terminal at
$HOME. If$LIM_HOSTis set and remote, it SSHs into that host as user$LIM_AI_USERwith a port forward for the viewer (ssh -L <port>:localhost:<port>); otherwise it opens a local shell.
You then run your coder alias in that terminal as usual.
Install:
make install (see Installing LIM) builds and installs the extension. To build or install it separately:
make vscode # builds and packages the extension
make install-vscode # installs the extension into VS Codemake install installs everything (binary, config files, and VS Code extension). To uninstall:
make uninstall # removes the binary and config files
make uninstall-all # removes installed files and uninstalls the extensionThen in VS Code, click the LIM rocket icon in the status bar (or Ctrl+Shift+P -- LIM: Open Workspace). This opens a terminal panel and waits for the browser server. Run your coder alias in that terminal, and the viewer will connect automatically once the server is ready (immediately, if the persistent server is already running).
To reset the browser viewer (close and reopen its tab), press Ctrl+2 or Ctrl+Shift+R -- the extension's LIM: Reload Browser command. It re-reads the tokenized URL, so it is also how you pick up a fresh token after a machine reboot or a server replacement by /reset.
To rebuild both the C++ binary and the extension:
makeLIM supports benchmarking modes controlled by LIM_CHATBOT_MODE to compare its persistent KV-cache approach against standard chatbot and cache-aware decoding. Each mode writes a TPS log (log/<N>.tps) recording context position and tokens-per-second at each speed diagnostic update (see Logging).
| Value | Mode | Description |
|---|---|---|
0 (default) |
LIM normal | KV-cache persists across turns. Each token is decoded once and never re-decoded. Same approach as llama-cli interactive mode. |
1 |
Standard chatbot | Emulates a plain text-based chatbot: the conversation is held as text, so each turn re-tokenizes the full conversation text and re-decodes everything from scratch (no KV reuse). TPS includes re-tokenize + re-decode + generation. |
2 |
Cache-aware prefix match | Emulates llama-server behavior: KV-cache stays in memory, but each turn re-tokenizes the full conversation text and compares against the cached prefix to find where to resume decoding; if the re-tokenization drifts from the sampled tokens, it re-decodes from the last prompt checkpoint. |
Chatbot modes (1 and 2) automatically enforce LIM_HONEST_SPEED=1, so the TPS in the logs includes re-decode overhead and reflects the true wall-clock cost of each approach.
Note: For fair comparisons, run benchmarks with an empty system prompt (remove
~/.config/lim/prompt) to inhibit tool calls. In mode 1, previous tool calls would be both re-decoded and re-executed on every turn, causing massive slowdowns plus unwanted side effects.
LIM can start SearXNG and Docling automatically when needed, but you must install them first:
cd ~
git clone https://github.com/searxng/searxng.git
cd searxng
pip install --no-build-isolation -e .Then edit searx/settings.yml: set a value for server.secret_key (required). The defaults for bind_address (127.0.0.1) and port (8888) are already correct for LIM. You may need to disable some search engines that block automated queries -- see the SearXNG docs for details. Override the start command with LIM_SEARXNG_CMD if installed elsewhere.
cd ~
python3 -m venv venv
source venv/bin/activate
pip3 install docling-serve
deactivateLIM expects the Docling binary at ~/venv/bin/docling-serve and starts it on port 5001. Override the start command with LIM_DOCLING_CMD if installed elsewhere.
The FIFO at $LIM_CONFIG_DIR/lim.fifo persists across sessions: it is owned by the persistent browser server (started on first use), which keeps the FIFO open for as long as it is running. If the node is removed while the server is alive, the server keeps reading the old, deleted inode and browser output goes nowhere -- but lim detects this: at startup and on /reset it compares the FIFO inode it writes to against the inodes the server holds open (via /proc/<pid>/fd), and when they differ it tears the stale server down and starts a fresh one on the current node. Afterwards, reconnect by loading the URL the new server's lim prints (the fresh server uses a fresh token). Ensure aiohttp is installed: pip3 install aiohttp.
LIM tries to connect to SearXNG on 127.0.0.1:8888. Make sure SearXNG is installed at ~/searxng/ and configured to listen on that port in settings.yml. You can also run SearXNG externally and point LIM to it -- just ensure the server is already listening before your first web search.
Ensure Docling is installed. If it's not at ~/venv/bin/docling-serve, set LIM_DOCLING_CMD to the correct command, e.g.:
export LIM_DOCLING_CMD="/opt/docling/bin/docling-serve run --enable-ui --host 127.0.0.1"If you don't have an NVIDIA GPU, build for CPU only:
make GGML_CUDA=off- The page is served by the persistent browser server, so it must be running before the page can load. If it is (the common case), just load or reload the viewer page using the URL lim printed (or
cat $LIM_CONFIG_DIR/lim.viewer_url), or press Ctrl+2 / Ctrl+Shift+R in VS Code (which re-reads the token). If it is not -- only right after a machine reboot -- start lim first; it starts the server on first need and prints the URL. - Check that nothing else is using port 8765 (or your custom
LIM_PORT) -- on the LIM host for the server, or locally when using a tunnel. - The server binds to
127.0.0.1only: from another machine, establish the tunnel (ssh -L 8765:localhost:8765 $LIM_AI_USER@$LIM_HOST) and load the URL lim printed. If the tunnel's local side uses a different port, setLIM_VIEWER_URL(the printed URL already uses it). - A page loaded without the token -- a bare
viewer.html, or a URL from before the last server start -- reports that it cannot connect: load the current URL lim prints, orcat $LIM_CONFIG_DIR/lim.viewer_url.
See Git Safe Directories in the User Setup section. In a directory the LLM created, the refusal is intentional (it keeps the LLM-owned gitdir inert).
Input history is persisted to .lim_history in the current working directory and survives across sessions; /cd and /load switch to the new directory's file (like a relaunch).
Each session writes a numbered log file in $LIM_LOG_DIR/<N>. With LIM_DEBUG=1, a companion token-level trace is written to $LIM_LOG_DIR/<N>.tokens showing every token fed into and generated by the model, with labels like FEED SYSTEM_PROMPT_INIT, FEED USER_INPUT, etc. A TPS log is also written to $LIM_LOG_DIR/<N>.tps recording context token count and tokens-per-second at each speed diagnostic update.
See LICENSE for details.
