[piyushmakhija28]

A stack sampler on a real pipeline run found the "ghost gap" — a 90–140 second stretch with zero logging that had been showing up between session tagging and the first checkpoint.

176.0s  3-level-flow.py:555  main              ← the whole run
 96.0s  preference_tracker.py:213  _read_file  ← here
 40.0s  supervised.py:98  _drain
  7.0s  preference_tracker.py:226  _collect_session_texts

_collect_session_texts read every .md/.txt/.json/.log file under every session directory, fully into memory, with no bound of any kind.

The directory is ~/.claude/memory/logs/sessions. On the machine that found this:

size	4.4 GB
session directories	1,658
matching text files, read in full, every run	11,050

The engine writes a session directory per run, so each run made the next one slower, permanently and without limit. One run had already been pushed past the entry point's 300-second orchestration cap.

The bound comes from what the reading is for

track_preferences lowercases the text and asks whether known skill and agent names appear in it, keeping anything seen in at least MIN_OCCURRENCES (3) sessions. That is a frequency heuristic over recent behaviour — a modest purpose for 4.4 GB of I/O.

The hundred-and-fifty-first megabyte of one log cannot establish a preference that three separate sessions did not, and a session from six months ago is not evidence about what the user prefers now.

So: the newest MAX_SESSIONS_SCANNED (25) directories by mtime, and at most MAX_FILE_BYTES (1 MB) per file. Sorted by mtime rather than by name because session directory names are UUID-bearing and do not sort chronologically — sorting by name would have dropped an arbitrary set rather than the oldest.

Measured, same machine and task

	before	after
full pipeline	174 s	51 s
the collection step alone	106 s	1.0 s

In that run the recency cap did all the work and the per-file cap never engaged — the expected shape, since the bulk is many sessions rather than one enormous file. The per-file cap is there for the other shape.

The cap reports itself

[preference_tracker] sampled history: skipped 1633 older session(s), truncated 0 file(s) at 1000000 bytes

This repository's rule is that a cap must not be silent: a sampled history that does not say it was sampled reads as a complete one. On stderr rather than through a logger because this module is loaded by path as a standalone script and has none.

Tests

Ten, each bound paired with a case proving the bound is what makes the difference — without the cap all 20 fixture sessions are read, with it 5 are, and the ones dropped are the oldest by mtime. A second pair proves the notice is not unconditional.

They assert on how much is read, not on what comes back. The function's output was always correct; correctness was never the problem, and no correctness test could have seen this.

A correction to my own earlier reports

I twice attributed this to the wrong place before measuring. First to session_loader — right neighbourhood, wrong mechanism, since that module scans nothing. Then to ~/.claude/logs/sessions at 684 MB / 47 directories — the wrong directory entirely; the scanned one is ~/.claude/memory/logs/sessions at 4.4 GB / 1,658. Both guesses came before the sampler. The sampler took one run.

🤖 Generated with Claude Code