From 8aa6d34fb0f5ae62b18c2cff1d2087ef581e1603 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 31 Oct 2026 21:54:10 +1100 Subject: [PATCH 01/14] fix(decorators): clear an L1-only cache whole on invalidate-all and let the memo own the kill A whole-function invalidation (cache_clear(), or invalidate_cache() with no arguments) deleted an L1-only cache's entries one by one, from the keys the process had recorded. A fill records its key only after its put, and a single-key invalidation untracks its key before its delete, so an entry stored before the call could outlive it and stay servable by the lock-free hit. The cache is now cleared whole. The memo now ends an entry's lock-free hits itself: forget, repoint and clear kill the entries they drop, so no removal route pairs a kill with a slot drop by hand. The at-fork reset clears the memo after it forgets every stored entry, which also ends an entry a refresh had in hand when a fork interrupted it, and a memo write skips a state the reset retired. A slot is read back with an exact type check, so an object planted in the memo's dict through the garbage collector is a miss or a TypeError, not undefined behaviour. L1-only TTLs run on the OS monotonic clock that the extension reads, so a patched time.monotonic does not move them. The docs say so and show how to test expiry instead. The memo's bounds are exported from the extension, the unused HitEntry getters and pyfunction shims are gone, and the SWR jitter's low end is one named constant. --- docs/api-reference.md | 2 +- docs/backends/none.md | 24 ++++ rust/src/l1_hit.rs | 216 ++++++++++++++++------------ rust/src/lib.rs | 7 +- src/cachekit/decorators/wrapper.py | 47 +++--- src/cachekit/object_cache.py | 96 ++++++++----- tests/unit/test_key_memo.py | 58 ++++---- tests/unit/test_l1_lock_free_hit.py | 134 ++++++++++++++--- 8 files changed, 369 insertions(+), 215 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index b61d7c3e..12ff3eab 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -82,7 +82,7 @@ def your_function(args): #### Core Parameters -- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. +- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`) times it on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). - **`namespace`** (`str | None`, default: `None`) - Cache key prefix for organization - **`serializer`** (`str | SerializerProtocol`, default: `"default"`) - Serializer name (`"default"`, `"std"`, `"auto"`, `"arrow"`, `"orjson"`) or `SerializerProtocol` instance - **`integrity_checking`** (`bool`, default: `True`; any non-bool raises `ConfigurationError`) - Enable xxHash3-64 checksums for corruption detection (non-cryptographic — detects bit rot and storage bugs, NOT tampering; tamper resistance requires encryption). Behaviour when a writer and reader disagree on this flag: [cross-config reads](serializers/auto.md#cross-config-reads-integrity_checking-mismatch) diff --git a/docs/backends/none.md b/docs/backends/none.md index 16b95805..69c3cf9e 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -202,6 +202,30 @@ get_origin() assert len(calls) == 2 # ran on both calls: nothing was stored ``` +## TTLs Follow the OS Monotonic Clock + +An L1-only entry expires on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry by moving fake time. `@cache.local()` expires its entries on the same clock. A cache with a backend is unchanged. + +To test what happens after an entry expires, remove it instead: + +```python +from cachekit import cache + +calls = [] + + +@cache(backend=None, ttl=60) +def get_rate(currency: str) -> float: + calls.append(currency) + return 1.08 + + +get_rate("EUR") +get_rate.invalidate_cache("EUR") # in place of moving time past ttl; cache_clear() removes every entry +get_rate("EUR") +assert calls == ["EUR", "EUR"] +``` + ## Characteristics - Latency: 0.31μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 22 times the instructions; in-memory, no network diff --git a/rust/src/l1_hit.rs b/rust/src/l1_hit.rs index fcbdab1e..ee2ef66c 100644 --- a/rust/src/l1_hit.rs +++ b/rust/src/l1_hit.rs @@ -2,21 +2,23 @@ //! //! `@cache(backend=None)` keeps its values in an `ObjectCache` (`src/cachekit/object_cache.py`), whose //! writes and removals run under its lock. A hit does not take it: [`HitMemo::get`] maps the call's -//! arguments ([`memo_key`]) to the [`HitEntry`] the cache stored for them, in one dict lookup, and +//! arguments ([`memo_key_of`]) to the [`HitEntry`] the cache stored for them, in one dict lookup, and //! serves the entry's value while the entry is alive and before its deadline, counting the hit on the //! function's `HitCounter`. //! //! The cache keeps the memo in step under its lock. An entry it stores or finds for a call is memoised -//! under that call's key, and an entry it removes (expiry, eviction, invalidation, clear, the at-fork -//! reset) is killed and forgotten before the removal returns. So a hit that starts after a removal has -//! returned never serves the removed value: it finds no memo slot, or it finds the entry dead. And no -//! removed value stays referenced from here, as the memo holds no removed entry. +//! under that call's key, and an entry it removes (expiry, eviction, invalidation, replacement, clear, +//! the at-fork reset) is forgotten before the removal returns; a refreshed entry's slots are repointed +//! at its successor. Forgetting, repointing or clearing kills the entries it drops, so the memo, not +//! each removal route, ends their lock-free hits. A hit that starts after a removal has returned never serves the removed +//! value: it finds no memo slot, or it finds the entry dead. And no removed value stays referenced +//! from here, as the memo holds no removed entry. //! //! Each entry lists the memo keys whose slots hold it, so a removal finds its slots without a scan, //! and the memo keeps every key list in step with its slots. The arguments a memo key holds are //! therefore kept only while their slot is: at most [`MEMO_MAX_ENTRIES`] keys per function. //! -//! Deadlines are read on one clock, [`monotonic`], which `ObjectCache` uses too. +//! Deadlines are read on one clock, [`now`] (`monotonic` in Python), which `ObjectCache` uses too. use std::ptr; use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU64, Ordering}; @@ -28,19 +30,23 @@ use pyo3::ffi; use pyo3::gc::PyVisit; use pyo3::intern; use pyo3::prelude::*; -use pyo3::types::{PyDict, PyString, PyTuple}; +use pyo3::types::{PyDict, PyString, PyTuple, PyType}; use pyo3::PyTraverseError; use crate::hit_counter::PyHitCounter; +// The memo's bounds, exported to Python under the same names for tests. + /// A call's str and bytes arguments and keyword names may hold at most this many characters or bytes -/// between them: the memo keeps what it is keyed by. Mirrors `_MEMO_MAX_LEN` in decorators/wrapper.py. -const MEMO_MAX_LEN: ffi::Py_ssize_t = 256; -/// Arguments per call, positional and keyword together (`_MEMO_MAX_ARGS`). -const MEMO_MAX_ARGS: usize = 8; -/// Memo slots per decorated function (`_MEMO_MAX_ENTRIES`). A full memo is emptied rather than -/// evicted slot by slot, so a hit writes nothing to it. -const MEMO_MAX_ENTRIES: usize = 1024; +/// between them, as the memo keeps what it is keyed by. That bounds an entry at a few KiB (a str of +/// 4-byte characters also keeps its UTF-8 copy once the key generator encodes it), and it bounds the +/// hashing a lookup repeats for strings that are new to each call. +pub(crate) const MEMO_MAX_LEN: ffi::Py_ssize_t = 256; +/// Arguments per call, positional and keyword together. +pub(crate) const MEMO_MAX_ARGS: usize = 8; +/// Memo slots per decorated function. A full memo is emptied rather than evicted slot by slot, so a +/// hit writes nothing to it. +pub(crate) const MEMO_MAX_ENTRIES: usize = 1024; // ── The clock ──────────────────────────────────────────────────────────────────────────────── @@ -54,7 +60,15 @@ static EPOCH: OnceLock = OnceLock::new(); /// reader that loaded the pointer before a replacement still holds a live object. static CLOCK_OVERRIDE: AtomicPtr = AtomicPtr::new(ptr::null_mut()); -fn now(py: Python<'_>) -> PyResult { +/// Seconds on the clock `ObjectCache` and the lock-free hit share, from an arbitrary zero. +/// +/// One clock for both, so an entry's deadline, computed by the cache, means the same instant to +/// the hit that compares against it. The OS monotonic clock is read here, not `time.monotonic`, so +/// patching `time.monotonic` (monkeypatch, freezegun) does not move it: the hit would have to call +/// back into Python on every call to follow a patch. +#[pyfunction] +#[pyo3(name = "monotonic")] +pub fn now(py: Python<'_>) -> PyResult { let fake = CLOCK_OVERRIDE.load(Ordering::Acquire); if fake.is_null() { return Ok(EPOCH.get_or_init(Instant::now).elapsed().as_secs_f64()); @@ -65,15 +79,6 @@ fn now(py: Python<'_>) -> PyResult { .extract() } -/// Seconds on the clock `ObjectCache` and the lock-free hit share, from an arbitrary zero. -/// -/// One clock for both, so an entry's deadline, computed by the cache, means the same instant to -/// the hit that compares against it. -#[pyfunction] -pub fn monotonic(py: Python<'_>) -> PyResult { - now(py) -} - /// Replace the shared clock with a callable returning seconds, or restore the real one with None. /// /// For tests only: the real clock cannot be moved, and a test of expiry or refresh needs both the @@ -135,18 +140,26 @@ unsafe fn type_of<'py>(py: Python<'py>, a: *mut ffi::PyObject) -> Bound<'py, PyA /// The memo key of a call's arguments, or None when the memo does not take the call. /// -/// The admission rules (`_memo_key` in decorators/wrapper.py documents them): every value an exact -/// int, float, bool, None, str or bytes; every keyword name an exact str; at most -/// [`MEMO_MAX_ARGS`] arguments; at most [`MEMO_MAX_LEN`] characters or bytes in the str and bytes -/// values and the keyword names together. The shapes, when [`admit`] needs no types: a lone int -/// or str positional argument itself, which hashes and compares without a tuple around it, and the -/// argument tuple itself otherwise (a lone None too, as None means no key). With types: `(t, a)` -/// and `(t, a, u, b)` for one and two positional arguments, `(args, types)` for more, and -/// `(args, types, items, item_types)` with keywords, sorted by name so keywords in any order share -/// one slot. No two shapes are equal: a lone argument is an int or a str, never a tuple; an untyped -/// tuple holds no type and no tuple; the two- and four-item typed keys start with a type, the -/// others with a tuple. -fn memo_key_of<'py>( +/// The lock-free hit looks this key up, and the decorator's slow path memoises its entry under it +/// (`_memo_key` in decorators/wrapper.py), so both compute one key. +/// +/// The admission rules: every value an exact int, float, bool, None, str or bytes; every keyword +/// name an exact str; at most `MEMO_MAX_ARGS` arguments; at most `MEMO_MAX_LEN` characters or bytes +/// in the str and bytes values and the keyword names together. Two equal values of one of these +/// types always produce the same cache key, as their MessagePack is the same (0.0 and -0.0 both +/// normalise to 0.0). A subclass (an `IntEnum`, a str subclass) or a container never matches, so it +/// takes the full key path on every call. +/// +/// The shapes, when [`admit`] needs no types: a lone int or str positional argument itself, which +/// hashes and compares without a tuple around it, and the argument tuple itself otherwise (a lone +/// None too, as None means no key). With types: `(t, a)` and `(t, a, u, b)` for one and two +/// positional arguments, `(args, types)` for more, and `(args, types, items, item_types)` with +/// keywords, sorted by name so keywords in any order share one slot. No two shapes are equal: a +/// lone argument is an int or a str, never a tuple; an untyped tuple holds no type and no tuple; the +/// two- and four-item typed keys start with a type, the others with a tuple. +#[pyfunction] +#[pyo3(name = "memo_key")] +pub fn memo_key_of<'py>( args: &Bound<'py, PyTuple>, kwargs: &Bound<'py, PyDict>, ) -> PyResult>> { @@ -260,25 +273,16 @@ fn memo_key_of<'py>( )) } -/// The memo key of a call's arguments, or None when the memo does not take the call. -/// -/// The key the lock-free hit looks up, for the decorator's slow path to memoise its entry under. -#[pyfunction] -pub fn memo_key<'py>( - args: &Bound<'py, PyTuple>, - kwargs: &Bound<'py, PyDict>, -) -> PyResult>> { - memo_key_of(args, kwargs) -} - // ── Entries and the memo ───────────────────────────────────────────────────────────────────── /// What a lock-free hit needs of one `ObjectCache` entry: its value, the time before which it may be /// served without the cache's lock, whether it is still stored, and when such a hit last read it. /// /// The value and deadline never change: the cache stores a new `HitEntry` when it refreshes an entry. -/// The cache kills an entry under its lock when it removes it, and a hit reads `alive` after it -/// found the entry, with acquire ordering, so a removal that has returned before a hit starts is seen. +/// The memo kills an entry, under the cache's lock, when the cache forgets it, repoints its slots or +/// clears the memo ([`HitMemo::forget`], [`HitMemo::repoint`], [`HitMemo::clear`]), and a hit reads +/// `alive` after it found the entry, with acquire ordering, so a removal that has returned before a +/// hit starts is seen. #[pyclass(frozen, module = "cachekit._rust_serializer")] pub struct HitEntry { value: Py, @@ -288,7 +292,9 @@ pub struct HitEntry { used_at: AtomicU64, /// The memo keys whose slots hold this entry. Only [`HitMemo`] writes it, under the cache's lock, /// and never calls into Python while it holds the mutex; the mutex makes the entry shareable - /// between threads and lets the garbage collector read the list. + /// between threads and lets the garbage collector read the list. No thread holds it at a fork, so + /// the at-fork hook's forget never waits on it: it is held only in Rust code with no point where + /// a thread can be paused, and `os.fork()` holds the GIL or, free-threaded, stops the world first. keys: Mutex>>, } @@ -301,6 +307,11 @@ impl HitEntry { fn take_keys(&self) -> Vec> { std::mem::take(&mut *self.keys()) } + + /// End the entry for lock-free hits. + fn kill(&self) { + self.alive.store(false, Ordering::Release); + } } #[pymethods] @@ -316,12 +327,7 @@ impl HitEntry { } } - /// End the entry for lock-free hits; the cache calls this, under its lock, on every removal. - fn kill(&self) { - self.alive.store(false, Ordering::Release); - } - - /// Whether the entry is still stored. + /// Whether the entry can still be served without the lock: false once the memo forgot it. #[getter] fn alive(&self) -> bool { self.alive.load(Ordering::Acquire) @@ -333,18 +339,6 @@ impl HitEntry { f64::from_bits(self.used_at.load(Ordering::Relaxed)) } - /// The time before which a lock-free hit may serve the value. - #[getter] - fn fresh_until(&self) -> f64 { - self.fresh_until - } - - /// The stored object, as a hit returns it. - #[getter] - fn value(&self, py: Python<'_>) -> Py { - self.value.clone_ref(py) - } - /// The memo keys whose slots hold the entry, as a snapshot. #[getter] fn memo_keys<'py>(&self, py: Python<'py>) -> PyResult> { @@ -368,14 +362,18 @@ impl HitEntry { /// One decorated function's memo: memo key -> the `HitEntry` the function's `ObjectCache` holds for /// that call, and the lock-free hit that reads it. /// -/// Only the cache writes the memo, under its lock (`set`, `forget`, `repoint`, `clear`), and each -/// write keeps the key lists of the entries it touches equal to their slots; `get` only reads it, -/// which CPython's dict allows from any thread. A memo answers only while the function's +/// Only the cache writes the memo, under its lock (`set`, `forget`, `repoint`, `clear`), and each write keeps +/// the key lists of the entries it touches equal to their slots; `get` only reads it, which +/// CPython's dict allows from any thread. A slot is read back as a `HitEntry` with a type check, not +/// on trust: code can reach the dict through the garbage collector (`gc.get_referents`) and plant +/// another object, which must cost a miss or an error, never memory safety. A memo answers only while the function's /// `__module__` and `__qualname__` are the objects it was built with, which its cache keys embed: /// renaming the function re-keys its calls, as the key generator reads the names on every call. #[pyclass(frozen, module = "cachekit._rust_serializer")] pub struct HitMemo { memo: Py, + /// `HitEntry`'s type, held so a hit checks a slot's type with one comparison. + entry_type: Py, counter: Py, func: Py, module: Py, @@ -520,7 +518,14 @@ impl HitMemo { let Some(found) = self.memo.bind(py).get_item(&key)? else { return Ok(miss.unbind()); }; - // SAFETY: set() stores HitEntry objects only. + // A slot holds a HitEntry unless code planted another object (see HitMemo): a miss. + if !ptr::eq( + unsafe { ffi::Py_TYPE(found.as_ptr()) }, + self.entry_type.as_ptr().cast(), + ) { + return Ok(miss.unbind()); + } + // SAFETY: found's type is exactly HitEntry, which cannot be subclassed. let entry = unsafe { found.cast_unchecked::() }.get(); if !entry.alive.load(Ordering::Acquire) { return Ok(miss.unbind()); @@ -534,6 +539,23 @@ impl HitMemo { } Ok(miss.unbind()) } + + /// Drop every slot and every entry's key list, and kill the entries too when `kill`. A full memo + /// empties without killing before it takes a new key: its entries stay stored, for the lock to + /// serve. + fn drop_slots(&self, py: Python<'_>, kill: bool) { + let memo = self.memo.bind(py); + for held in memo.values() { + if let Ok(held) = held.cast_exact::() { + let held = held.get(); + if kill { + held.kill(); + } + drop(held.take_keys()); + } + } + memo.clear(); + } } /// [`HitMemo::lookup`] as a builtin function bound to its memo: `HitMemo.get` makes one, and the @@ -606,6 +628,7 @@ impl HitMemo { let descriptors = NameDescriptors::of(func.bind(py)); Self { memo: PyDict::new(py).unbind(), + entry_type: py.get_type::().unbind(), counter, func, module, @@ -648,8 +671,7 @@ impl HitMemo { match memo.get_item(key)? { Some(held) if held.is(entry) => return Ok(()), Some(held) => { - // SAFETY: the memo holds HitEntry objects only. - let held = unsafe { held.cast_unchecked::() }.get(); + let held = held.cast_exact::()?.get(); let mut kept = Vec::new(); let mut failed = None; for k in held.take_keys() { @@ -667,7 +689,7 @@ impl HitMemo { return Err(e); } } - None if memo.len() >= MEMO_MAX_ENTRIES => self.clear(py), + None if memo.len() >= MEMO_MAX_ENTRIES => self.drop_slots(py, false), None => {} } memo.set_item(key, entry)?; @@ -675,10 +697,16 @@ impl HitMemo { Ok(()) } - /// Drop every slot that holds `entry`, as the cache removes it. Call under the cache's lock. + /// Kill `entry`, then drop every slot that holds it, as the cache removes it. Call under the + /// cache's lock. + /// + /// The kill comes first, so a hit that found the entry before its slots go reads it dead, and + /// the slots go next, so the memo holds no removed value. fn forget(&self, py: Python<'_>, entry: &Bound<'_, HitEntry>) -> PyResult<()> { + let entry_ref = entry.get(); + entry_ref.kill(); let memo = self.memo.bind(py); - for key in entry.get().take_keys() { + for key in entry_ref.take_keys() { let key = key.bind(py); if memo.get_item(key)?.is_some_and(|held| held.is(entry)) { memo.del_item(key)?; @@ -687,8 +715,11 @@ impl HitMemo { Ok(()) } - /// Point every slot that holds `stale` at `fresh`, as the cache refreshes an entry. Call under - /// the cache's lock. + /// Point every slot that holds `stale` at `fresh`, then kill `stale`, as the cache refreshes an + /// entry. Call under the cache's lock. + /// + /// A hit that finds either entry reads it alive or misses, and none serves the stale value once + /// this returns, a failed write included. fn repoint( &self, py: Python<'_>, @@ -697,25 +728,21 @@ impl HitMemo { ) -> PyResult<()> { let memo = self.memo.bind(py); let keys = stale.get().take_keys(); - for key in &keys { - memo.set_item(key.bind(py), fresh)?; - } + let moved = keys + .iter() + .try_for_each(|key| memo.set_item(key.bind(py), fresh)); fresh.get().keys().extend(keys); - Ok(()) + stale.get().kill(); + moved } - /// Drop every slot, and with them every entry's key list. Call under the cache's lock. + /// Kill every entry the memo holds and drop every slot. Call under the cache's lock. + /// + /// The cache's clear and the at-fork reset call this after they forget every stored entry, so it + /// also ends an entry no stored entry points at: one a refresh or a memo write had in hand when + /// a fork interrupted it. fn clear(&self, py: Python<'_>) { - let memo = self.memo.bind(py); - for held in memo.values() { - // SAFETY: the memo holds HitEntry objects only. - drop( - unsafe { held.cast_unchecked::() } - .get() - .take_keys(), - ); - } - memo.clear(); + self.drop_slots(py, true); } /// The memo keys with a slot, for tests. @@ -729,6 +756,7 @@ impl HitMemo { fn __traverse__(&self, visit: PyVisit<'_>) -> Result<(), PyTraverseError> { visit.call(&self.memo)?; + visit.call(&self.entry_type)?; visit.call(&self.counter)?; visit.call(&self.func)?; visit.call(&self.module)?; diff --git a/rust/src/lib.rs b/rust/src/lib.rs index d5c9ce8e..83c6525f 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -56,9 +56,12 @@ fn _rust_serializer(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> { // The memoised L1-only hit, its entries and their clock (decorators/wrapper.py, object_cache.py) m.add_class::()?; m.add_class::()?; - m.add_function(wrap_pyfunction!(l1_hit::memo_key, m)?)?; - m.add_function(wrap_pyfunction!(l1_hit::monotonic, m)?)?; + m.add_function(wrap_pyfunction!(l1_hit::memo_key_of, m)?)?; + m.add_function(wrap_pyfunction!(l1_hit::now, m)?)?; m.add_function(wrap_pyfunction!(l1_hit::set_clock, m)?)?; + m.add("MEMO_MAX_LEN", l1_hit::MEMO_MAX_LEN)?; + m.add("MEMO_MAX_ARGS", l1_hit::MEMO_MAX_ARGS)?; + m.add("MEMO_MAX_ENTRIES", l1_hit::MEMO_MAX_ENTRIES)?; // ByteStorage.retrieve() envelope-verification-failure taxonomy. Same // __module__ patch as KeyringConfigurationError below: create_exception! sets it to the diff --git a/src/cachekit/decorators/wrapper.py b/src/cachekit/decorators/wrapper.py index 3e310036..0422fe69 100644 --- a/src/cachekit/decorators/wrapper.py +++ b/src/cachekit/decorators/wrapper.py @@ -11,7 +11,7 @@ import os import threading import time -from collections.abc import AsyncIterator, Callable, Iterator +from collections.abc import AsyncIterator, Callable, Hashable, Iterator from enum import Enum from typing import TYPE_CHECKING, Any, NamedTuple, TypeVar, Union @@ -611,27 +611,9 @@ def logger(): ) -# The memo (create_cache_wrapper's HitMemo) takes a call only when every argument's exact type is int, -# float, bool, NoneType, str or bytes. Two equal values of one of these types always produce the same -# key: their MessagePack is the same, 0.0 and -0.0 included, since both normalise to 0.0. Types are -# tested by identity, as the key generator's fast path does, so an argument's type is never hashed or -# compared through its metaclass. A subclass (IntEnum, a str subclass) or a container never matches, -# so it takes the full key path on every call. -# The memo keeps what it is keyed by, keyword names included, so a call's str and bytes arguments and -# keyword names may hold at most _MEMO_MAX_LEN characters or bytes between them. That bounds an entry -# at a few KiB: a str of 4-byte characters also keeps its UTF-8 copy once the key generator encodes -# it. It also bounds the hashing a lookup repeats for strings that are new to each call. -# _memo_key, the memo key of a call's arguments or None, is the extension's memo_key -# (rust/src/l1_hit.rs), so the lock-free hit and the slow path that memoises compute one key. A call -# of only ints, strs and Nones is keyed by its argument tuple itself: those never compare equal -# across types. A float or bool (1, 1.0 and True are equal) or a bytes puts every type in the key. -# These mirror the extension's constants. -_MEMO_MAX_LEN = 256 -# Arguments per call, positional and keyword together. -_MEMO_MAX_ARGS = 8 -# Slots per decorated function. A full memo is emptied rather than evicted slot by slot, so a hit -# writes nothing to it. -_MEMO_MAX_ENTRIES = 1024 +# _memo_key, the memo key of a call's arguments or None when the memo (create_cache_wrapper's HitMemo) +# does not take the call, is the extension's memo_key, so the lock-free hit and the slow path that +# memoises compute one key. rust/src/l1_hit.rs documents which calls it takes and the memo's bounds. # What HitMemo.get returns when it cannot answer a call. _MISS = object() @@ -1892,9 +1874,7 @@ def _delete_then_forget(cache_keys: list[str] | None) -> InvalidationOutcome | N finally: _forget_flights(cache_keys) - def _l1_only_fill( - cache_key: str, call_args: tuple[Any, ...], call_kwargs: dict[str, Any], memo_key: tuple[Any, ...] | None - ) -> Any: + def _l1_only_fill(cache_key: str, call_args: tuple[Any, ...], call_kwargs: dict[str, Any], memo_key: Hashable | None) -> Any: """Run the function for an L1-only miss and store its result, memoised under memo_key (see _call_key). A miss is a call that runs it.""" assert _object_cache is not None # noqa: S101 - L1-only miss path only @@ -1905,7 +1885,7 @@ def _l1_only_fill( return result async def _l1_only_fill_async( - cache_key: str, call_args: tuple[Any, ...], call_kwargs: dict[str, Any], memo_key: tuple[Any, ...] | None + cache_key: str, call_args: tuple[Any, ...], call_kwargs: dict[str, Any], memo_key: Hashable | None ) -> Any: """_l1_only_fill for a coroutine function.""" assert _object_cache is not None # noqa: S101 - L1-only miss path only @@ -1915,7 +1895,7 @@ async def _l1_only_fill_async( _cached_keys.add((_l2_scope(), cache_key)) return result - def _call_key(call_args: tuple[Any, ...], call_kwargs: dict[str, Any]) -> tuple[str, tuple[Any, ...] | None]: + def _call_key(call_args: tuple[Any, ...], call_kwargs: dict[str, Any]) -> tuple[str, Hashable | None]: """A call's cache key, and for an L1-only function with a memo the memo key to memoise its entry under, or None. @@ -3261,7 +3241,8 @@ def _local_invalidate_all() -> None: while this runs, stays in _cached_keys for the next attempt. Keys other processes wrote and this one never saw stay in L2 until their TTL. Another tenant's entry keeps its L2 value and stays tracked; only its L1 copy is evicted, because L1 is not - tenant-scoped. + tenant-scoped. An L1-only function's ObjectCache is cleared whole, so every entry stored + before this call goes, recorded or not. Failed deletes are logged once per call with their count: the call never raises, so this record is the caller's only signal, and a per-key record would flood during an @@ -3287,10 +3268,14 @@ def _local_invalidate_all() -> None: _cached_keys.discard(entry) if entry in watch: # rewritten meanwhile: its new value may still be in L2 _cached_keys.add(entry) - if _object_cache: - _object_cache.delete(key) - elif _l1_cache: + if _object_cache is None and _l1_cache: _l1_cache.invalidate(key) + if _object_cache: + # An L1-only function's cache goes whole, not key by key from the snapshot: a fill + # records its key only after its put, and a single-key invalidation untracks its key + # before its delete, so the snapshot can miss an entry stored before this call. L1 + # is not tenant-scoped, and an L1-only function has nothing in L2. + _object_cache.clear() if fallbacks: # Once per call, like the ERROR below. A backend whose multi-key delete always fails # (e.g. an ACL that denies UNLINK) otherwise silently pays one round trip per key. diff --git a/src/cachekit/object_cache.py b/src/cachekit/object_cache.py index d55a3056..fe6aabfe 100644 --- a/src/cachekit/object_cache.py +++ b/src/cachekit/object_cache.py @@ -15,13 +15,17 @@ import threading import weakref from collections import OrderedDict, deque -from collections.abc import Iterable +from collections.abc import Hashable, Iterable from dataclasses import dataclass from typing import Any, cast from cachekit._rust_serializer import HitEntry, HitMemo from cachekit._rust_serializer import monotonic as _monotonic +# The SWR threshold's ±10% jitter. Its low end also bounds a lock-free hit: short of it no read can +# flag a refresh, so the hit may serve an entry up to there without the lock. +_JITTER_LOW, _JITTER_HIGH = 0.9, 1.1 + def _estimate_object_size(obj: Any) -> int: """Best-effort byte-size estimate of a Python object graph. @@ -129,16 +133,14 @@ def remove(self, key: str) -> None: the memo stay consistent. Anti-resurrection needs no per-key residue: a refresh can only land on the exact entry (generation) it was started against. - A lock-free hit of the entry ends before the entry leaves the store. The kill comes first, so - a hit that found the entry before its memo slots go reads it dead, and the slots go next, so - the memo holds no removed value. + A lock-free hit of the entry ends before the entry leaves the store: the memo's forget kills + the entry, then drops its slots (see HitMemo.forget). """ entry = self.store.get(key) if entry is None: return hit = entry.hit if hit is not None: - hit.kill() memo = self.memo assert memo is not None # noqa: S101 - an entry has a hit only in a cache with a memo memo.forget(hit) @@ -215,13 +217,19 @@ def evict_lru(self) -> None: return def kill_all(self) -> None: - """End every entry's lock-free hits at once: kill them all, then empty the memo; call with lock held.""" + """End every entry's lock-free hits: forget every stored entry, then clear the memo; call with lock held. + + Every stored entry is killed, memoised or not: a thread that forked inside a critical section + returns there in the child on the state this retires and may memoise one of its entries + before _take_back_if_retired forgets it, and a hit in between must find it dead. The memo's + clear then kills what it still holds: an entry a refresh or a memo write had in hand when a + fork interrupted it, which no stored entry points at yet or any more. + """ memo = self.memo if memo is None: return - for entry in list(self.store.values()): - if entry.hit is not None: - entry.hit.kill() + for entry in list(self.store.values()): # a snapshot: a finalizer may re-enter and write + memo.forget(cast("HitEntry", entry.hit)) memo.clear() @@ -253,7 +261,9 @@ class ObjectCache: Times are read on ``cachekit._rust_serializer.monotonic``, the clock a lock-free hit (below) reads too, so a deadline means the same instant to - both. With a memo, an entry's LRU recency is the clock reading of its latest + both. The extension reads the OS monotonic clock itself, so patching + ``time.monotonic`` (monkeypatch, freezegun) does not move this cache's + deadlines. With a memo, an entry's LRU recency is the clock reading of its latest use: exact wherever the clock advances between two uses, as the OS monotonic clock it reads does within nanoseconds. Of two entries last used at one reading, the one stored first is evicted first. @@ -282,10 +292,12 @@ class ObjectCache: its hard expiry, or with ``swr=True`` the earliest point ``get_with_swr`` could flag it for refresh (``ttl * swr_threshold_ratio * 0.9`` after it was stored, with the ttl it was stored with). Every removal (expiry, eviction, - ``delete``, replacement, ``clear``, the at-fork reset) kills the entry's - ``HitEntry`` and drops its memo slots before it returns, so no lock-free hit - serves a removed entry once its removal has returned, and the memo keeps no - removed value. ``hits`` counts lookups through the lock only. + ``delete``, replacement, ``clear``, the at-fork reset) has the memo forget + the entry's ``HitEntry`` before it returns, and a refresh has it repoint the + entry's slots at the new one: either way the memo kills the old ``HitEntry`` + and holds it no more. So no lock-free hit serves a removed entry once its + removal has returned, and the memo keeps no removed value. ``hits`` counts + lookups through the lock only. Fork safety: in a forked child an at-fork hook (``_reset_caches_after_fork``) repairs every live cache before the child's first call. The parent's threads @@ -361,10 +373,11 @@ def __init__( self._max_size_bytes = max_size_bytes self._swr_threshold_ratio = swr_threshold_ratio self._swr_retry_interval = swr_retry_interval - # The part of an entry's ttl a lock-free hit may serve it for: up to the jitter's low end of the - # SWR threshold, short of which get_with_swr never flags a refresh, or all of it. - self._fresh_ratio = swr_threshold_ratio * 0.9 if swr else 1.0 - self._memo = memo + # The part of an entry's ttl short of which no read can flag a refresh: the SWR threshold at its + # jitter's low end. get_with_swr serves an entry that young without drawing a jitter. + self._swr_floor = swr_threshold_ratio * _JITTER_LOW + # The part of an entry's ttl a lock-free hit may serve it for: up to that floor, or all of it. + self._fresh_ratio = self._swr_floor if swr else 1.0 # Every critical section binds this once: `s = self._state; with s.lock: ...`. Reading # self._state again inside one would mix states if the at-fork hook replaced it meanwhile. self._state = _ObjectCacheState(memo) @@ -380,7 +393,7 @@ def __init__( # Public API # ------------------------------------------------------------------ - def get(self, key: str, memo_key: Any = None) -> tuple[bool, Any]: + def get(self, key: str, memo_key: Hashable | None = None) -> tuple[bool, Any]: """Retrieve a value from the cache. Expired entries are removed on read (lazy expiry). @@ -408,6 +421,10 @@ def get(self, key: str, memo_key: Any = None) -> tuple[bool, Any]: self._misses += 1 return False, None + # A read's bookkeeping, the same in get_with_swr: its recency (the store's order, or the + # stamp the memo cache's records read), its memo slot, and the hit. A read path that skips + # a part breaks LRU or the counts. Inline, not a helper: a call costs every @cache.local() + # hit about 9% more instructions. entry.used_at = now # most recently used if s.memo is None: s.store.move_to_end(key) @@ -436,7 +453,7 @@ def peek(self, key: str) -> tuple[bool, Any]: return False, None return True, entry.value - def get_with_swr(self, key: str, ttl: float, memo_key: Any = None) -> tuple[bool, Any, bool, int]: + def get_with_swr(self, key: str, ttl: float, memo_key: Hashable | None = None) -> tuple[bool, Any, bool, int]: """Get value with stale-while-revalidate support. Once an entry is older than ``ttl * swr_threshold_ratio`` (±10% jitter @@ -473,7 +490,7 @@ def get_with_swr(self, key: str, ttl: float, memo_key: Any = None) -> tuple[bool self._misses += 1 return False, None, False, 0 - entry.used_at = now + entry.used_at = now # a read's bookkeeping, as in get if s.memo is None: s.store.move_to_end(key) elif memo_key is not None: @@ -482,14 +499,14 @@ def get_with_swr(self, key: str, ttl: float, memo_key: Any = None) -> tuple[bool version = entry.generation age = now - entry.cached_at - threshold = ttl * self._swr_threshold_ratio - # Not stale even at the jitter's low end (0.9), so no draw from random could flag it: - # the common hit returns here without one. - if age <= threshold * 0.9: + # Not stale even at the jitter's low end, so no draw from random could flag it: the + # common hit returns here without one. + if age <= ttl * self._swr_floor: return True, entry.value, False, version needs_refresh = False - # ±10% jitter staggers refreshes when many keys cross the threshold together - jitter = random.uniform(0.9, 1.1) # noqa: S311 - not cryptographic + threshold = ttl * self._swr_threshold_ratio + # The jitter staggers refreshes when many keys cross the threshold together + jitter = random.uniform(_JITTER_LOW, _JITTER_HIGH) # noqa: S311 - not cryptographic backing_off = entry.refresh_failed_at is not None and now - entry.refresh_failed_at < self._swr_retry_interval if age > threshold * jitter and key not in s.refreshing and not backing_off: s.refreshing[key] = entry.generation @@ -555,14 +572,12 @@ def complete_refresh(self, key: str, version: int, value: Any, ttl: float) -> bo entry.refresh_failed_at = None # success ends any retry back-off stale = entry.hit if stale is not None: - # A HitEntry's value never changes, so the refresh gets a new one. Its memo slots - # point at the new one before the stale one is killed: a lock-free hit that finds - # either reads it alive or misses, and none serves the stale value after this returns. + # A HitEntry's value never changes, so the refresh gets a new one, and the memo + # repoints the stale one's slots at it, then kills the stale one (HitMemo.repoint). memo = s.memo assert memo is not None # noqa: S101 - an entry has a hit only in a cache with a memo fresh = entry.hit = HitEntry(value, now + ttl * self._fresh_ratio, now) memo.repoint(stale, fresh) - stale.kill() self._take_back_if_retired(s, fresh) else: s.store.move_to_end(key) @@ -608,7 +623,7 @@ def fail_refresh(self, key: str, version: int) -> None: entry = s.store[key] entry.refresh_failed_at = _monotonic() - def put(self, key: str, value: Any, ttl: int, memo_key: Any = None) -> None: + def put(self, key: str, value: Any, ttl: int, memo_key: Hashable | None = None) -> None: """Store a value in the cache. When a bound is exceeded: @@ -657,7 +672,7 @@ def put(self, key: str, value: Any, ttl: int, memo_key: Any = None) -> None: used_at=now, size_bytes=size, generation=self._generation, - hit=None if self._memo is None else HitEntry(value, now + ttl * self._fresh_ratio, now), + hit=None if s.memo is None else HitEntry(value, now + ttl * self._fresh_ratio, now), ) s.store[key] = entry s.size_bytes += size @@ -745,10 +760,14 @@ def max_size_bytes(self) -> int | None: # Private helpers # ------------------------------------------------------------------ - def _memoise(self, s: _ObjectCacheState, entry: _Entry, memo_key: Any) -> None: - """Point memo_key's memo slot at the entry, for the next such call's lock-free hit; call with s.lock held.""" + def _memoise(self, s: _ObjectCacheState, entry: _Entry, memo_key: Hashable) -> None: + """Point memo_key's memo slot at the entry, for the next such call's lock-free hit; call with s.lock held. + + Not for an entry of a state the at-fork reset retired: the reset killed it, and its slot would + only displace one the child set meanwhile. + """ hit = entry.hit - if hit is None: + if hit is None or self._state is not s: return memo = s.memo assert memo is not None # noqa: S101 - an entry has a hit only in a cache with a memo @@ -760,10 +779,11 @@ def _take_back_if_retired(self, s: _ObjectCacheState, hit: HitEntry) -> None: A thread that forked inside its critical section, from a signal handler or a finalizer, returns there in the child on the state the at-fork hook retired, after the hook emptied the - memo. A slot it then set for an entry of that state would outlive every removal. + memo. A slot it then set for an entry of that state would outlive every removal. _memoise + sets none on a state already retired, but a fork during the memo write itself (a finalizer + that an allocation in it runs) retires the state after that check. """ if self._state is not s: - hit.kill() cast("HitMemo", s.memo).forget(hit) def _evict(self, s: _ObjectCacheState, extra_bytes: int, need_slot: bool) -> None: diff --git a/tests/unit/test_key_memo.py b/tests/unit/test_key_memo.py index 30b3a67f..c0916cbd 100644 --- a/tests/unit/test_key_memo.py +++ b/tests/unit/test_key_memo.py @@ -21,9 +21,10 @@ from hypothesis import strategies as st from cachekit import cache +from cachekit._rust_serializer import MEMO_MAX_ARGS, MEMO_MAX_ENTRIES, MEMO_MAX_LEN from cachekit.decorators import wrapper as wrapper_module from cachekit.decorators.stats_context import get_current_function_stats -from cachekit.decorators.wrapper import _MEMO_MAX_ARGS, _MEMO_MAX_ENTRIES, _MEMO_MAX_LEN, _memo_key +from cachekit.decorators.wrapper import _memo_key from cachekit.key_generator import CacheKeyGenerator from tests.unit.test_key_registry import _closure_cell @@ -131,21 +132,24 @@ def _key_used(fn: Any, args: tuple[Any, ...], kwargs: dict[str, Any]) -> str | N class _Gate: - """An L1-only function's cache whose delete waits for the test, so an invalidation can be held - between deciding to erase an entry and erasing it while the test makes another call.""" + """An L1-only function's cache whose delete and clear wait for the test, so an invalidation (of + one key, or of the whole function) can be held between deciding to erase entries and erasing + them while the test makes another call.""" def __init__(self, fn: Any) -> None: self.deleting = threading.Event() self.release = threading.Event() oc = _closure_cell(fn, "_object_cache").cell_contents - delete = oc.delete - def gated(key: str) -> bool: - self.deleting.set() - assert self.release.wait(10), "the test never released the delete" - return delete(key) + def held(remove: Any) -> Any: + def gated(*args: Any) -> Any: + self.deleting.set() + assert self.release.wait(10), "the test never released the delete" + return remove(*args) - oc.delete = gated + return gated + + oc.delete, oc.clear = held(oc.delete), held(oc.clear) def _gate(fn: Any) -> _Gate: @@ -204,8 +208,8 @@ def __str__(self) -> str: _Int(1), Decimal(1), frozenset(), - "x" * (_MEMO_MAX_LEN + 1), - b"x" * (_MEMO_MAX_LEN + 1), + "x" * (MEMO_MAX_LEN + 1), + b"x" * (MEMO_MAX_LEN + 1), ] @@ -225,20 +229,20 @@ def test_only_exact_admitted_types_are_memoised(self, value: Any) -> None: assert _memo_key((), {"x": value}) is None def test_admitted_types_at_the_length_cap_are_memoised(self) -> None: - for value in (0, -1, 2**63, 1.5, -0.0, True, None, "", "x" * _MEMO_MAX_LEN, b"x" * _MEMO_MAX_LEN): + for value in (0, -1, 2**63, 1.5, -0.0, True, None, "", "x" * MEMO_MAX_LEN, b"x" * MEMO_MAX_LEN): assert _memo_key((value,), {}) is not None def test_a_str_subclass_keyword_name_is_not_memoised(self) -> None: assert _memo_key((), {_Str("x"): 1}) is None def test_strings_and_keyword_names_share_one_length_budget(self) -> None: - half, rest = _MEMO_MAX_LEN // 2, _MEMO_MAX_LEN - _MEMO_MAX_LEN // 2 + half, rest = MEMO_MAX_LEN // 2, MEMO_MAX_LEN - MEMO_MAX_LEN // 2 for extra in (0, 1): fits = extra == 0 assert (_memo_key(("x" * half, "y" * (rest + extra)), {}) is not None) is fits assert (_memo_key((b"x" * half, 1, "y" * (rest + extra)), {}) is not None) is fits assert (_memo_key(("x" * half,), {"y" * (rest - 1 + extra): "z"}) is not None) is fits - assert (_memo_key((), {"x": "y" * (_MEMO_MAX_LEN - 2 + extra), "n": 1}) is not None) is fits + assert (_memo_key((), {"x": "y" * (MEMO_MAX_LEN - 2 + extra), "n": 1}) is not None) is fits def test_types_are_tested_by_identity_never_hashed_or_compared(self) -> None: """A metaclass that defines __eq__ makes a class unhashable; one that hashes and compares @@ -250,9 +254,9 @@ def test_types_are_tested_by_identity_never_hashed_or_compared(self) -> None: assert _memo_key((), {"x": value}) is None def test_at_most_max_args_arguments_are_memoised(self) -> None: - assert _memo_key(tuple(range(_MEMO_MAX_ARGS)), {}) is not None - assert _memo_key(tuple(range(_MEMO_MAX_ARGS + 1)), {}) is None - assert _memo_key(tuple(range(_MEMO_MAX_ARGS)), {"x": 1}) is None + assert _memo_key(tuple(range(MEMO_MAX_ARGS)), {}) is not None + assert _memo_key(tuple(range(MEMO_MAX_ARGS + 1)), {}) is None + assert _memo_key(tuple(range(MEMO_MAX_ARGS)), {"x": 1}) is None def test_shapes_never_collide(self) -> None: calls = [((1,), {}), ((1, 2), {}), ((1, 2, 3), {}), ((), {}), ((1,), {"b": 2}), ((1, 2), {"b": 2})] @@ -284,7 +288,7 @@ class TestKeyBytesUnchanged: ((1, 2, 3), {}, True), (("q",), {"limit": 10}, True), (("q",), {"limit": 10, "offset": 5}, True), - (("x" * (_MEMO_MAX_LEN + 1),), {}, False), + (("x" * (MEMO_MAX_LEN + 1),), {}, False), ((_Color.RED,), {}, False), (([1, 2],), {}, False), ], @@ -312,7 +316,7 @@ def fn(*a: Any, **k: Any) -> int: # Values that compare equal across types (1, 1.0, True; 0, 0.0, -0.0, False), values just past # the length cap, and a few the memo never takes, so equal-but-different calls interleave. _ARGS = st.sampled_from( - [0, 1, 1.0, True, False, 0.0, -0.0, None, "", "1", b"1", "x" * _MEMO_MAX_LEN, "x" * (_MEMO_MAX_LEN + 1), _Color.RED, [1]] + [0, 1, 1.0, True, False, 0.0, -0.0, None, "", "1", b"1", "x" * MEMO_MAX_LEN, "x" * (MEMO_MAX_LEN + 1), _Color.RED, [1]] ) @settings(max_examples=200, deadline=None) @@ -474,9 +478,9 @@ def test_the_memo_stays_bounded(self) -> None: def fn(x: int) -> int: return x - for i in range(_MEMO_MAX_ENTRIES + 10): + for i in range(MEMO_MAX_ENTRIES + 10): fn(i) - assert 0 < len(_memo(fn)) <= _MEMO_MAX_ENTRIES + assert 0 < len(_memo(fn)) <= MEMO_MAX_ENTRIES def test_a_miss_still_runs_inside_the_stats_context(self) -> None: seen: list[Any] = [] @@ -607,7 +611,7 @@ async def run() -> None: def test_a_full_memo_at_every_cap_stays_small(self) -> None: """Every call at every cap: eight keyword names and eight values of 4-byte characters, sharing - _MEMO_MAX_LEN between them, the most a call can make the memo keep. docs/performance.md + MEMO_MAX_LEN between them, the most a call can make the memo keep. docs/performance.md states this bound: 3.7 to 4.0MiB measured on CPython 3.10, 3.12, 3.14 and 3.14t.""" @cache(backend=None, encryption=False) @@ -615,12 +619,12 @@ def fn(**kwargs: Any) -> int: return 1 wide = "\U0001f600" - each = _MEMO_MAX_LEN // (2 * _MEMO_MAX_ARGS) - for i in range(_MEMO_MAX_ENTRIES): + each = MEMO_MAX_LEN // (2 * MEMO_MAX_ARGS) + for i in range(MEMO_MAX_ENTRIES): tag = f"{i:04x}" - fn(**{tag + chr(97 + j) + wide * (each - 5): tag + wide * (each - 4) for j in range(_MEMO_MAX_ARGS)}) + fn(**{tag + chr(97 + j) + wide * (each - 5): tag + wide * (each - 4) for j in range(MEMO_MAX_ARGS)}) memo = _memo(fn) - assert len(memo) == _MEMO_MAX_ENTRIES + assert len(memo) == MEMO_MAX_ENTRIES assert _deep_size(memo.keys()) < 4.5 * 2**20 # what the memo keeps beyond the cache's entries def test_keyword_names_past_the_cap_cache_without_the_memo(self) -> None: @@ -632,7 +636,7 @@ def fn(**kwargs: Any) -> int: calls += 1 return 1 - name = "x" * (_MEMO_MAX_LEN + 1) + name = "x" * (MEMO_MAX_LEN + 1) for _ in range(3): fn(**{name: 1}) assert calls == 1 diff --git a/tests/unit/test_l1_lock_free_hit.py b/tests/unit/test_l1_lock_free_hit.py index 222ab299..d465ff97 100644 --- a/tests/unit/test_l1_lock_free_hit.py +++ b/tests/unit/test_l1_lock_free_hit.py @@ -1,9 +1,10 @@ """The L1-only hit answered without a lock: cachekit._rust_serializer.HitMemo (ObjectCache, "Lock-free hits"). -Pins what the hit must never do: serve an entry once its removal has returned, by any of the five routes -(expiry, eviction, invalidation, cache_clear, the at-fork reset); keep a removed value referenced from the -extension; judge a deadline on another clock than ObjectCache's; evict out of LRU order; or key a call -differently from the memo-key rules the Python implementation had. +Pins what the hit must never do: serve an entry once its removal has returned, by any route (expiry, +eviction, invalidation, replacement, cache_clear, the at-fork reset, and a refresh, which replaces the +entry's HitEntry); keep a removed value referenced from the extension; judge a deadline on another clock +than ObjectCache's; evict out of LRU order; or key a call differently from the memo-key rules the Python +implementation had. """ from __future__ import annotations @@ -28,9 +29,9 @@ from hypothesis import strategies as st from cachekit import cache, object_cache -from cachekit._rust_serializer import HitCounter, HitMemo, monotonic +from cachekit._rust_serializer import MEMO_MAX_ARGS, MEMO_MAX_ENTRIES, MEMO_MAX_LEN, HitCounter, HitMemo, monotonic from cachekit.config import L1CacheConfig -from cachekit.decorators.wrapper import _MEMO_MAX_ARGS, _MEMO_MAX_ENTRIES, _MEMO_MAX_LEN, _memo_key +from cachekit.decorators.wrapper import _memo_key from cachekit.object_cache import ObjectCache from tests.unit.test_key_registry import _closure_cell from tests.utils.clock import fake_clock, shared_clock @@ -177,6 +178,42 @@ def fn(x: int) -> _Value: fn(1) assert calls == [1, 2, 1] + @pytest.mark.parametrize("clear", ["cache_clear", "invalidate_cache"]) + def test_a_whole_function_invalidation_removes_an_entry_it_has_no_record_of(self, clear: str) -> None: + """A fill records its key only after its put, and a single-key invalidation untracks its key + before its delete, so a whole-function invalidation cannot work from those records: an entry + stored before it must not outlive it, recorded or not.""" + + @cache(backend=None, ttl=300) + def fn(x: int) -> _Value: + return _Value(x) + + oc, memo = _cache_and_memo(fn) + unrecorded = _Value("unrecorded") + ref = weakref.ref(unrecorded) + oc.put("no record names this key", unrecorded, 300, memo_key=_memo_key((1,), {})) # a fill before its record + assert fn(1) is unrecorded # served without the lock + del unrecorded + getattr(fn, clear)() + assert len(memo) == 0 + assert _gone(ref) + assert fn(1).n == 1 + + def test_replacement(self) -> None: + """A put over a stored key replaces its entry, and the old entry's slots go with it whatever memo + key holds them: the replacing put is memoised under another key here, so it cannot re-point + the old slot and hide a replacement that skipped the removal.""" + oc, memo = _memo_cache(max_entries=8) + old = _Value("old") + ref = weakref.ref(old) + stale = _put_and_hit_entry(oc, "a", old) + key_b = _memo_key(("b",), {}) + oc.put("key:a", _Value("new"), 3600, memo_key=key_b) + assert (stale.alive, _hit(memo, "a"), memo.keys()) == (False, _MISS, [key_b]) + assert _hit(memo, "b").n == "new" + del old, stale + assert _gone(ref) + @pytest.mark.skipif(not hasattr(os, "fork"), reason="fork() not available on this platform") def test_the_at_fork_reset(self) -> None: """A parent thread holds the cache's lock at fork, so the child's cache starts empty: its memo @@ -257,8 +294,8 @@ def test_a_full_memo_takes_its_entries_key_lists_with_it(self) -> None: """A full memo empties before it takes a new key, and the entries it held drop their key lists with it: an entry keeps a call's arguments only while that call has a slot, so the memo bounds what all of them keep.""" - oc, memo = _memo_cache(max_entries=4 * _MEMO_MAX_ENTRIES) - for i in range(_MEMO_MAX_ENTRIES + 100): + oc, memo = _memo_cache(max_entries=4 * MEMO_MAX_ENTRIES) + for i in range(MEMO_MAX_ENTRIES + 100): _put_and_hit_entry(oc, f"k{i}", i) assert len(memo) == 100 assert _keys_held_by_entries(oc) == Counter(memo.keys()) @@ -274,18 +311,47 @@ def test_a_slot_moved_to_another_entry_leaves_the_first(self) -> None: assert len(memo) == 0 assert first.alive # the entry that lost its slot stays stored, for the lock to serve - def test_a_slot_set_on_a_state_the_fork_reset_retired_is_taken_back(self) -> None: + def test_no_slot_is_set_for_an_entry_of_a_state_the_fork_reset_retired(self) -> None: """A thread that forks inside its critical section (from a signal handler or a finalizer) returns - there in the child on the state the at-fork hook retired. A slot it then sets would point at an - entry no removal will ever reach, so the cache ends that entry and drops the slot.""" + there in the child on the state the at-fork hook retired. A slot it set for an entry of that + state would point at an entry no removal will ever reach, and would displace the slot the + child set for the same call meanwhile, so the cache sets none.""" oc, memo = _memo_cache(max_entries=8) oc.put("key:k", _Value("k"), 3600) # stored without a slot s = oc._state entry = s.store["key:k"] + key = _memo_key(("k",), {}) with s.lock: oc._reset_after_fork() # what the hook does in a child forked inside this section - oc._memoise(s, entry, _memo_key(("k",), {})) - assert (len(memo), entry.hit.alive, _hit(memo, "k")) == (0, False, _MISS) + oc.put("key:k", _Value("child"), 3600, memo_key=key) # the child, before the section resumes + oc._memoise(s, entry, key) + assert (len(memo), entry.hit.alive, _hit(memo, "k").n) == (1, False, "child") + + def test_a_fork_inside_a_refresh_leaves_the_child_no_live_slot(self, monkeypatch: pytest.MonkeyPatch) -> None: + """A refresh stores its new HitEntry on the entry, then has the memo repoint the old one's slots. + A fork between the two runs the at-fork reset while the old HitEntry holds slots that no stored + entry points at; the reset must end it all the same, or a child that never returns to the + refresh (a multiprocessing child) serves it.""" + clock = fake_clock(monkeypatch) + oc, memo = _memo_cache(max_entries=8) + stale = _put_and_hit_entry(oc, "k", _Value("old"), ttl=100) + clock.monotonic = lambda: 1060.0 + _, _, needs_refresh, version = oc.get_with_swr("key:k", 100, _memo_key(("k",), {})) + assert needs_refresh + seen: list[tuple[int, bool, Any]] = [] + + class ForkingMemo: # the memo, with a fork just before the refresh's repoint + def repoint(self, *args: Any) -> None: + oc._reset_after_fork() + seen.append((len(memo), stale.alive, _hit(memo, "k"))) # what a child that never returns sees + memo.repoint(*args) + + def __getattr__(self, name: str) -> Any: + return getattr(memo, name) + + oc._state.memo = ForkingMemo() # type: ignore[assignment] + oc.complete_refresh("key:k", version, _Value("new"), ttl=100) + assert seen == [(0, False, _MISS)] def test_a_refresh_on_a_state_the_fork_reset_retired_leaves_no_slot(self, monkeypatch: pytest.MonkeyPatch) -> None: clock = fake_clock(monkeypatch) @@ -315,6 +381,18 @@ def test_a_slot_left_on_a_removed_entry_still_misses(self) -> None: memo.set(_memo_key(("k",), {}), entry_hit) # what no cache does: a slot on a removed entry assert _hit(memo, "k") is _MISS + def test_an_object_planted_in_a_slot_is_a_miss_not_a_crash(self) -> None: + """Python code can reach the memo's dict through gc.get_referents and put any object in a slot. + The extension reads a slot back with a type check, so the hit misses and a memo write over the + slot raises; neither reads the object as a HitEntry.""" + oc, memo = _memo_cache(max_entries=8) + _put(oc, "k", _Value("k")) + (slots,) = [o for o in gc.get_referents(memo) if type(o) is dict] + slots[_memo_key(("k",), {})] = _Value("planted") + assert _hit(memo, "k") is _MISS + with pytest.raises(TypeError): + oc.get("key:k", _memo_key(("k",), {})) # memoises its entry over the planted slot + def _keys_held_by_entries(oc: ObjectCache) -> Counter[Any]: """The memo keys the stored entries list as theirs: equal to the memo's keys, if the cache kept both in step.""" @@ -430,7 +508,7 @@ def _reference_memo_key(args: tuple[Any, ...], kwargs: dict[str, Any]) -> tuple[ if n == 1: a = args[0] t = type(a) - if t is int or ((t is str or t is bytes) and len(a) <= _MEMO_MAX_LEN) or t is float or t is bool or a is None: + if t is int or ((t is str or t is bytes) and len(a) <= MEMO_MAX_LEN) or t is float or t is bool or a is None: return (t, a) return None if n == 2: @@ -438,16 +516,16 @@ def _reference_memo_key(args: tuple[Any, ...], kwargs: dict[str, Any]) -> tuple[ t = type(a) u = type(b) if t is int or t is float or t is bool or a is None: - if u is int or ((u is str or u is bytes) and len(b) <= _MEMO_MAX_LEN) or u is float or u is bool or b is None: + if u is int or ((u is str or u is bytes) and len(b) <= MEMO_MAX_LEN) or u is float or u is bool or b is None: return (t, a, u, b) elif t is str or t is bytes: if u is int or u is float or u is bool or b is None: - if len(a) <= _MEMO_MAX_LEN: + if len(a) <= MEMO_MAX_LEN: return (t, a, u, b) - elif (u is str or u is bytes) and len(a) + len(b) <= _MEMO_MAX_LEN: + elif (u is str or u is bytes) and len(a) + len(b) <= MEMO_MAX_LEN: return (t, a, u, b) return None - if len(args) + len(kwargs) > _MEMO_MAX_ARGS: + if len(args) + len(kwargs) > MEMO_MAX_ARGS: return None size = 0 for a in args: @@ -456,13 +534,13 @@ def _reference_memo_key(args: tuple[Any, ...], kwargs: dict[str, Any]) -> tuple[ return None size += cost if not kwargs: - return (args, tuple(map(type, args))) if size <= _MEMO_MAX_LEN else None + return (args, tuple(map(type, args))) if size <= MEMO_MAX_LEN else None for name, a in kwargs.items(): cost = _reference_memo_size(a) if type(name) is not str or cost < 0: return None size += len(name) + cost - if size > _MEMO_MAX_LEN: + if size > MEMO_MAX_LEN: return None items = tuple(sorted(kwargs.items())) return (args, tuple(map(type, args)), items, tuple([type(value) for _, value in items])) @@ -486,7 +564,7 @@ class _Str(str): st.sampled_from([_Color.RED, _Str("a"), Decimal(1), [1], (1,), frozenset()]), ) _CALLS = st.tuples( - st.lists(_VALUES, max_size=_MEMO_MAX_ARGS + 1), + st.lists(_VALUES, max_size=MEMO_MAX_ARGS + 1), st.dictionaries(st.sampled_from(["a", "b", "c", "x" * 100, _Str("s")]), _VALUES, max_size=3), ) @@ -694,6 +772,18 @@ def fn(k: int) -> tuple[int, bytes]: created.append(value) return value + oc, _ = _cache_and_memo(fn) + + def cleared() -> None: + """fn.cache_clear(), then every entry stored before it began is stamped removed: whatever way + the clear empties the cache, none of them may be served once it has returned.""" + with oc._state.lock: + live = [entry.value for entry in oc._state.store.values()] + fn.cache_clear() + at = tick() + for value in live: + removed_at.setdefault(id(value), at) + stop = threading.Event() violations: list[tuple[int, int, int]] = [] served = itertools.count() @@ -714,7 +804,7 @@ def invalidator(seed: int) -> None: rng = random.Random(seed) while not stop.is_set(): if rng.random() < 0.05: - fn.cache_clear() + cleared() else: fn.invalidate_cache(rng.randrange(16)) From 14739758458ea9ce436b39b5e78287af6420e452 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 31 Oct 2026 22:31:40 +1100 Subject: [PATCH 02/14] fix(decorators): cache_clear() clears every entry, on a function without parameters too cache_clear() called invalidate_cache() with no arguments, which on a function without parameters deletes only the entry of the call without arguments. A key= that reads its context files that call under many keys, so the other contexts' entries outlived the clear. cache_clear() now takes the whole-function path directly, after the preamble every invalidation runs (settle a zero-config function, resolve a lazy backend, apply the interop guard), which invalidate_cache() and ainvalidate_cache() now share. An L1-only fill no longer records its key in _cached_keys. Nothing reads the record once the whole-function path clears the cache whole, and it grew by one entry for every distinct key ever filled. Docs: the monotonic-clock note now names a bare @cache that resolves no backend and the stale-while-revalidate timing, and the free-threading audit row and the at-fork comments match the code. --- docs/api-reference.md | 2 +- docs/backends/none.md | 2 +- docs/free-threading.md | 2 +- src/cachekit/decorators/wrapper.py | 63 +++++++++++++---------------- src/cachekit/object_cache.py | 5 ++- tests/unit/test_l1_lock_free_hit.py | 33 +++++++++++++-- 6 files changed, 64 insertions(+), 43 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 12ff3eab..0489cf00 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -82,7 +82,7 @@ def your_function(args): #### Core Parameters -- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`) times it on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). +- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`, or a bare `@cache` that resolves no backend) times it, and its stale-while-revalidate refresh, on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). - **`namespace`** (`str | None`, default: `None`) - Cache key prefix for organization - **`serializer`** (`str | SerializerProtocol`, default: `"default"`) - Serializer name (`"default"`, `"std"`, `"auto"`, `"arrow"`, `"orjson"`) or `SerializerProtocol` instance - **`integrity_checking`** (`bool`, default: `True`; any non-bool raises `ConfigurationError`) - Enable xxHash3-64 checksums for corruption detection (non-cryptographic — detects bit rot and storage bugs, NOT tampering; tamper resistance requires encryption). Behaviour when a writer and reader disagree on this flag: [cross-config reads](serializers/auto.md#cross-config-reads-integrity_checking-mismatch) diff --git a/docs/backends/none.md b/docs/backends/none.md index 69c3cf9e..e58ddef6 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -204,7 +204,7 @@ assert len(calls) == 2 # ran on both calls: nothing was stored ## TTLs Follow the OS Monotonic Clock -An L1-only entry expires on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry by moving fake time. `@cache.local()` expires its entries on the same clock. A cache with a backend is unchanged. +An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for a bare `@cache` that finds no backend configured, as it runs in this mode. A cache with a backend is unchanged. To test what happens after an entry expires, remove it instead: diff --git a/docs/free-threading.md b/docs/free-threading.md index f275607c..2648427d 100644 --- a/docs/free-threading.md +++ b/docs/free-threading.md @@ -71,7 +71,7 @@ covered, plus what the free-threaded CI lane surfaced and sites added since: | `backends/redis/provider.py` `with_timeout` windows (`_open_windows`: pool → configured `socket_timeout` and the open windows' timeouts) | Per-PID `Lock` from `_pid_lock` (`_windows_locks`) around each window's open and close and around `listener_pool()`'s read of the pool's kwargs | Safe. Opening and closing each read and write the record and the pool's `socket_timeout` together, and a window can open and close on any thread's event loop over one shared pool, so without the lock a window closing the last record could drop one another thread just joined. No `await` runs under the lock. A child gets its own lock, as in the row above. A child forked while a window is open inherits that window but not the task holding it, so it could never close it: an `os.register_at_fork` hook drops every inherited record and puts each pool's configured timeout back, and each record carries the PID that opened it, so a fork made from C, which runs no hook, drops it on the pool's next window or listener clone instead; until then such a child opens its connections at the inherited window's timeout (no per-operation PID check, as in the object-cache row; `--py-call-uwsgi-fork-hooks` avoids it). A close restores the pool's timeout before it drops the record, so a fork between the two steps still leaves the hook a record to reset from. Close-order and fork tests (single-threaded) in `tests/unit/test_invalidation_listener.py` | | `decorators/wrapper.py` `_cached_keys` | Builtin `set`, snapshot-copied before iteration in invalidation | Safe — single ops atomic; a copy racing an add can only miss a concurrently-written key, which invalidate-all semantics tolerate | | `object_cache.py` + `_rust_serializer.HitMemo` lock-free L1-only hit (the [hit path](performance.md#decorator--l1-hit-hot-path) of an L1-only function) | No lock on a hit: `HitMemo.get` looks the call's memo key up in the function's memo, a builtin `dict` of `HitEntry` objects, then loads the entry's `alive` flag (acquire) and compares its deadline, then stores its use time (relaxed) and counts the hit on `HitCounter`. Only the function's `ObjectCache` writes the memo, under its lock: a put, or a read under the lock, memoises its entry, and every removal (expiry, eviction, `delete`, replacement, `clear`, the at-fork reset) kills the entry (release) and then drops its memo slots before it releases the lock. Each `HitEntry` lists the memo keys whose slots hold it, behind a `Mutex` that only those writes take and that is never held across Python code | Safe. A hit that starts after a removal returned, in the order the threads' own synchronisation gives them (a lock, an event, a join), finds no slot or finds the entry dead, so it never serves the removed value; a hit racing the removal may serve the value it found alive, as the removal had not returned. No removed value stays referenced from the memo. A refresh points the slots at the new entry before it kills the stale one. Two hits racing on one entry store either use time, which moves its LRU position by at most the time between them. `tests/unit/test_l1_lock_free_hit.py` pins each removal route and runs hitters, fillers and invalidators on shared keys under a 1 MB budget, so entries are evicted all along: no value served after its removal returned, on 3.14 and on 3.14t with the GIL off | -| `object_cache.py` `ObjectCache` + `_reset_caches_after_fork` | `RLock` on every public method. The entries, their byte total, their expiry and recency records, their in-flight refresh markers and the lock form one `_ObjectCacheState`, which every critical section binds once; the memo of an L1-only function's cache (the row above) is written only under that lock. An `os.register_at_fork` hook walks a `WeakSet` of live caches in a forked child | Safe. The hook runs while the child is single-threaded, so it takes no lock that can block and starts no thread. A cache whose lock was free at fork keeps its entries and loses its in-flight refresh markers, since the threads running those refreshes did not survive the fork, so the child's next stale read refreshes again. A cache whose lock was held kills every entry's lock-free hit, empties its memo and gets a fresh, empty state in one attribute store: a parent thread that held it is gone, and a forking thread that forked inside a critical section (from a signal handler or a finalizer) may never return there, as in a `multiprocessing` child. A forking thread that does return finishes on the state it bound, and takes back any memo slot it then sets for an entry of that state. The generation counter stays on the cache, so no refresh token is reused across states. A child forked without at-fork hooks (uWSGI without `--py-call-uwsgi-fork-hooks`) gets no repair: its first call hangs if a parent thread held that cache's lock at fork, and a key whose refresh was in flight is served stale until it expires. The check that could catch that, an owner-PID check, would cost a `getpid()` on every cached call. A sync function refreshes no key there at all (the SWR row). Tests in `tests/unit/test_object_cache.py::TestObjectCacheFork` | +| `object_cache.py` `ObjectCache` + `_reset_caches_after_fork` | `RLock` on every public method. The entries, their byte total, their expiry and recency records, their in-flight refresh markers and the lock form one `_ObjectCacheState`, which every critical section binds once; the memo of an L1-only function's cache (the row above) is written only under that lock. An `os.register_at_fork` hook walks a `WeakSet` of live caches in a forked child | Safe. The hook runs while the child is single-threaded, so it takes no lock that can block and starts no thread. A cache whose lock was free at fork keeps its entries and loses its in-flight refresh markers, since the threads running those refreshes did not survive the fork, so the child's next stale read refreshes again. A cache whose lock was held kills every entry's lock-free hit, empties its memo and gets a fresh, empty state in one attribute store: a parent thread that held it is gone, and a forking thread that forked inside a critical section (from a signal handler or a finalizer) may never return there, as in a `multiprocessing` child. A forking thread that does return finishes on the state it bound: it sets no memo slot for an entry of that state, and takes back one that a fork during the memo write left. The generation counter stays on the cache, so no refresh token is reused across states. A child forked without at-fork hooks (uWSGI without `--py-call-uwsgi-fork-hooks`) gets no repair: its first call hangs if a parent thread held that cache's lock at fork, and a key whose refresh was in flight is served stale until it expires. The check that could catch that, an owner-PID check, would cost a `getpid()` on every cached call. A sync function refreshes no key there at all (the SWR row). Tests in `tests/unit/test_object_cache.py::TestObjectCacheFork` | | `reliability/metrics_collection.py` `get_async_metrics_collector` | Double-checked module-global singleton | Safe — single-assignment publication of a fully-constructed object; worst case a benign duplicate worker-restart check | | `reliability/metrics_collection.py` `MetricsCollector` + `get_all_metrics` + `_metrics_lock()` | Module dicts of counters and gauges (`cache_operations_total`, `circuit_breaker_state`, ...) behind one `RLock` per PID, created with `dict.setdefault` | **Fixed** — every read and write takes the lock. A child gets its own, so a lock a parent thread held at a fork from C, with no at-fork hook run, is never waited on; before it was one module `RLock`, so a hookless child's first health check or `StructuredLogger.cache_operation()` could hang. The child's counts start from the values it inherited. Tested in `tests/unit/test_structured_logging.py` (`TestEntriesAfterFork`) | | `reliability/async_metrics.py` `_metrics_cache` + `_metrics_cache_lock()` | Double-checked module dict of Prometheus metric objects; one `Lock` per PID, created with `dict.setdefault`; held across `os.fork` by at-fork hooks | Safe — the lock-free read sees `None` or a fully constructed metric (single-assignment publication). `setdefault` is atomic under per-object locking, so threads racing to create a new process's lock all get the same one; a child forked without the hooks never takes its parent's lock. Tests in `tests/unit/test_async_metrics_shared_series.py` | diff --git a/src/cachekit/decorators/wrapper.py b/src/cachekit/decorators/wrapper.py index 0422fe69..d8938e29 100644 --- a/src/cachekit/decorators/wrapper.py +++ b/src/cachekit/decorators/wrapper.py @@ -19,6 +19,10 @@ from .. import invalidation, l1_cache from .._rust_serializer import HitCounter, HitMemo + +# _memo_key, the memo key of a call's arguments or None when the memo (create_cache_wrapper's HitMemo) +# does not take the call, is the extension's memo_key, so the lock-free hit and the slow path that +# memoises compute one key. rust/src/l1_hit.rs documents which calls it takes and the memo's bounds. from .._rust_serializer import memo_key as _memo_key from ..backends.errors import BackendError, UnsupportedTenantError from ..cache_handler import ( @@ -611,9 +615,6 @@ def logger(): ) -# _memo_key, the memo key of a call's arguments or None when the memo (create_cache_wrapper's HitMemo) -# does not take the call, is the extension's memo_key, so the lock-free hit and the slow path that -# memoises compute one key. rust/src/l1_hit.rs documents which calls it takes and the memo's bounds. # What HitMemo.get returns when it cannot answer a call. _MISS = object() @@ -1876,12 +1877,12 @@ def _delete_then_forget(cache_keys: list[str] | None) -> InvalidationOutcome | N def _l1_only_fill(cache_key: str, call_args: tuple[Any, ...], call_kwargs: dict[str, Any], memo_key: Hashable | None) -> Any: """Run the function for an L1-only miss and store its result, memoised under memo_key (see - _call_key). A miss is a call that runs it.""" + _call_key). A miss is a call that runs it. The key is not recorded in _cached_keys: an L1-only + function has nothing in L2, and its whole-function invalidation clears the cache whole.""" assert _object_cache is not None # noqa: S101 - L1-only miss path only _stats.record_miss() result = func(*call_args, **call_kwargs) _object_cache.put(cache_key, result, ttl=ttl if ttl is not None else 31536000, memo_key=memo_key) - _cached_keys.add((_l2_scope(), cache_key)) return result async def _l1_only_fill_async( @@ -1892,7 +1893,6 @@ async def _l1_only_fill_async( _stats.record_miss() result = await func(*call_args, **call_kwargs) _object_cache.put(cache_key, result, ttl=ttl if ttl is not None else 31536000, memo_key=memo_key) - _cached_keys.add((_l2_scope(), cache_key)) return result def _call_key(call_args: tuple[Any, ...], call_kwargs: dict[str, Any]) -> tuple[str, Hashable | None]: @@ -3271,10 +3271,8 @@ def _local_invalidate_all() -> None: if _object_cache is None and _l1_cache: _l1_cache.invalidate(key) if _object_cache: - # An L1-only function's cache goes whole, not key by key from the snapshot: a fill - # records its key only after its put, and a single-key invalidation untracks its key - # before its delete, so the snapshot can miss an entry stored before this call. L1 - # is not tenant-scoped, and an L1-only function has nothing in L2. + # An L1-only function records no keys (_l1_only_fill), so its cache goes whole: L1 is + # not tenant-scoped, and it has nothing in L2. _object_cache.clear() if fallbacks: # Once per call, like the ERROR below. A backend whose multi-key delete always fails @@ -3394,12 +3392,9 @@ def _invalidate_keys(cache_keys: list[str]) -> InvalidationOutcome: invalidation.publish(_backend, _registry_id, None if custom_key_func is not None else cache_keys[0]) return outcome - def invalidate_cache(*args: Any, **kwargs: Any) -> InvalidationOutcome | None: - """Delete the entry for these arguments from L1 and L2; with no arguments on a function that - takes parameters, every entry of the function. Never raises on a backend failure. - - Returns the call's InvalidationOutcome for one entry, and None for every entry. - """ + def _ready_to_invalidate() -> None: + """What every invalidation does first: settle a zero-config function, resolve a backed function's + lazy backend, and apply the interop guard reads and writes apply.""" nonlocal _backend if _zero_config_pending: @@ -3420,6 +3415,14 @@ def invalidate_cache(*args: Any, **kwargs: Any) -> InvalidationOutcome | None: if interop is not None: ensure_interop_backend_compatible(_backend) + def invalidate_cache(*args: Any, **kwargs: Any) -> InvalidationOutcome | None: + """Delete the entry for these arguments from L1 and L2; with no arguments on a function that + takes parameters, every entry of the function. Never raises on a backend failure. + + Returns the call's InvalidationOutcome for one entry, and None for every entry. + """ + _ready_to_invalidate() + # Fix #59: When called with no args on a parameterized function, # invalidate ALL cached entries for this function. # Without this, it generates a key for zero-arg call (never cached) → no-op. @@ -3436,23 +3439,7 @@ def invalidate_cache(*args: Any, **kwargs: Any) -> InvalidationOutcome | None: async def ainvalidate_cache(*args: Any, **kwargs: Any) -> InvalidationOutcome | None: """Async invalidate_cache: the same deletes, run off the event loop, and the same return value.""" - nonlocal _backend - - if _zero_config_pending: - _settle_zero_config() - - # L1-ONLY MODE: Skip backend lookup entirely - # This fixes the sentinel problem: when backend=None is explicitly passed, - # we should NOT try to get a backend from the provider - if not _l1_only_mode and _backend is None: - try: - _backend = _resolve_lazy_backend() - except Exception as e: - # If backend creation fails, can't invalidate L2 - _logger.debug("Failed to get backend for async invalidation: %s", redact_error_for_log(e)) - - if interop is not None: # interop guard, as in invalidate_cache - ensure_interop_backend_compatible(_backend) + _ready_to_invalidate() # Fix #59: When called with no args on a parameterized function, # invalidate ALL cached entries for this function. @@ -3513,7 +3500,12 @@ def cache_info() -> CacheInfo: return _stats.get_info() def cache_clear() -> None: - """Clear cache statistics and invalidate all cached entries.""" + """Clear cache statistics and invalidate all cached entries. + + All of them on a function without parameters too, where invalidate_cache() with no arguments + deletes only the entry of the call without arguments, though a key= that reads its context can + file that call under many keys. + """ _stats.clear() if _zero_config_pending: _settle_zero_config() @@ -3523,7 +3515,8 @@ def cache_clear() -> None: raise TypeError( "cache_clear() cannot clear cache for async functions with a backend. Use 'await fn.ainvalidate_cache()' instead." ) - invalidate_cache() + _ready_to_invalidate() + _delete_then_forget(None) # Other processes' invalidations reach this function's L1 through the process's listener. Only a # backed wrapper with an L1 registers: in L1-only mode nothing is shared, so nothing is announced. diff --git a/src/cachekit/object_cache.py b/src/cachekit/object_cache.py index fe6aabfe..4bdacdfa 100644 --- a/src/cachekit/object_cache.py +++ b/src/cachekit/object_cache.py @@ -222,8 +222,9 @@ def kill_all(self) -> None: Every stored entry is killed, memoised or not: a thread that forked inside a critical section returns there in the child on the state this retires and may memoise one of its entries before _take_back_if_retired forgets it, and a hit in between must find it dead. The memo's - clear then kills what it still holds: an entry a refresh or a memo write had in hand when a - fork interrupted it, which no stored entry points at yet or any more. + clear then drops every slot left, which already leaves no hit a way to an entry a refresh or a + memo write had in hand when a fork interrupted it; it kills such an entry too, as defence in + depth. """ memo = self.memo if memo is None: diff --git a/tests/unit/test_l1_lock_free_hit.py b/tests/unit/test_l1_lock_free_hit.py index d465ff97..1b619654 100644 --- a/tests/unit/test_l1_lock_free_hit.py +++ b/tests/unit/test_l1_lock_free_hit.py @@ -10,6 +10,7 @@ from __future__ import annotations import asyncio +import contextvars import enum import gc import itertools @@ -21,6 +22,7 @@ import time import weakref from collections import Counter, OrderedDict +from collections.abc import Callable from decimal import Decimal from typing import Any @@ -199,6 +201,31 @@ def fn(x: int) -> _Value: assert _gone(ref) assert fn(1).n == 1 + def test_cache_clear_removes_every_entry_of_a_function_without_parameters(self) -> None: + """On a function without parameters, invalidate_cache() with no arguments deletes the entry of + the call without arguments, but a key= that reads its context files that call under many keys: + cache_clear() removes every one, not only the clearing context's.""" + tenant: contextvars.ContextVar[str] = contextvars.ContextVar("tenant") + calls: list[str] = [] + + @cache(backend=None, ttl=300, key=lambda: f"cfg:{tenant.get()}") + def config() -> str: + calls.append(tenant.get()) + return tenant.get() + + def in_tenant(name: str, call: Callable[[], Any]) -> Any: + def run() -> Any: + tenant.set(name) + return call() + + return contextvars.copy_context().run(run) + + assert (in_tenant("a", config), in_tenant("b", config)) == ("a", "b") + in_tenant("a", config.cache_clear) + assert _cache_and_memo(config)[0].size == 0 + assert in_tenant("b", config) == "b" + assert calls == ["a", "b", "b"] + def test_replacement(self) -> None: """A put over a stored key replaces its entry, and the old entry's slots go with it whatever memo key holds them: the replacing put is memoised under another key here, so it cannot re-point @@ -327,11 +354,11 @@ def test_no_slot_is_set_for_an_entry_of_a_state_the_fork_reset_retired(self) -> oc._memoise(s, entry, key) assert (len(memo), entry.hit.alive, _hit(memo, "k").n) == (1, False, "child") - def test_a_fork_inside_a_refresh_leaves_the_child_no_live_slot(self, monkeypatch: pytest.MonkeyPatch) -> None: + def test_a_fork_inside_a_refresh_leaves_the_old_entry_dead(self, monkeypatch: pytest.MonkeyPatch) -> None: """A refresh stores its new HitEntry on the entry, then has the memo repoint the old one's slots. A fork between the two runs the at-fork reset while the old HitEntry holds slots that no stored - entry points at; the reset must end it all the same, or a child that never returns to the - refresh (a multiprocessing child) serves it.""" + entry points at. Emptying the memo already leaves the child no slot to reach it; the reset also + kills it, as defence in depth, so the child holds no live entry it dropped.""" clock = fake_clock(monkeypatch) oc, memo = _memo_cache(max_entries=8) stale = _put_and_hit_entry(oc, "k", _Value("old"), ttl=100) From 5f9d1829674a5a0546edaafeda5825a43a8318f7 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 31 Oct 2026 23:05:24 +1100 Subject: [PATCH 03/14] fix(decorators): a backed cache_clear() without parameters deletes its key and twin again The previous commit sent every cache_clear() down the whole-function path. On a backed function without parameters that path reaches only the keys this process recorded and, on Redis, the key registry, so the call's own key and its pre-0.20.0 twin could stay in L2 and keep being served. The whole-function path now applies to L1-only functions only; a backed function's cache_clear() goes through invalidate_cache() as it did before. Docs: an L1-only cache is any decorator that finds no backend configured, not only a bare @cache, and the cache_clear() and invalidate-all docstrings describe each mode. --- docs/api-reference.md | 2 +- docs/backends/none.md | 2 +- src/cachekit/decorators/wrapper.py | 25 ++++++++++++++----------- tests/unit/test_invalidate_no_args.py | 18 ++++++++++++++++++ tests/unit/test_l1_lock_free_hit.py | 7 ++++--- 5 files changed, 38 insertions(+), 16 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 0489cf00..6a668a14 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -82,7 +82,7 @@ def your_function(args): #### Core Parameters -- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`, or a bare `@cache` that resolves no backend) times it, and its stale-while-revalidate refresh, on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). +- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`, or any decorator that finds no backend configured, per [Nothing Configured](backends/README.md#4-nothing-configured-l1-only)) times it, and its stale-while-revalidate refresh, on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). - **`namespace`** (`str | None`, default: `None`) - Cache key prefix for organization - **`serializer`** (`str | SerializerProtocol`, default: `"default"`) - Serializer name (`"default"`, `"std"`, `"auto"`, `"arrow"`, `"orjson"`) or `SerializerProtocol` instance - **`integrity_checking`** (`bool`, default: `True`; any non-bool raises `ConfigurationError`) - Enable xxHash3-64 checksums for corruption detection (non-cryptographic — detects bit rot and storage bugs, NOT tampering; tamper resistance requires encryption). Behaviour when a writer and reader disagree on this flag: [cross-config reads](serializers/auto.md#cross-config-reads-integrity_checking-mismatch) diff --git a/docs/backends/none.md b/docs/backends/none.md index e58ddef6..fbf56773 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -204,7 +204,7 @@ assert len(calls) == 2 # ran on both calls: nothing was stored ## TTLs Follow the OS Monotonic Clock -An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for a bare `@cache` that finds no backend configured, as it runs in this mode. A cache with a backend is unchanged. +An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for any decorator that finds no backend configured, as it runs in this mode ([Nothing Configured](README.md#4-nothing-configured-l1-only)). A cache with a backend is unchanged. To test what happens after an entry expires, remove it instead: diff --git a/src/cachekit/decorators/wrapper.py b/src/cachekit/decorators/wrapper.py index d8938e29..f8f5ca66 100644 --- a/src/cachekit/decorators/wrapper.py +++ b/src/cachekit/decorators/wrapper.py @@ -3241,8 +3241,8 @@ def _local_invalidate_all() -> None: while this runs, stays in _cached_keys for the next attempt. Keys other processes wrote and this one never saw stay in L2 until their TTL. Another tenant's entry keeps its L2 value and stays tracked; only its L1 copy is evicted, because L1 is not - tenant-scoped. An L1-only function's ObjectCache is cleared whole, so every entry stored - before this call goes, recorded or not. + tenant-scoped. An L1-only function records no keys (_l1_only_fill); its ObjectCache is + cleared whole. Failed deletes are logged once per call with their count: the call never raises, so this record is the caller's only signal, and a per-key record would flood during an @@ -3500,23 +3500,26 @@ def cache_info() -> CacheInfo: return _stats.get_info() def cache_clear() -> None: - """Clear cache statistics and invalidate all cached entries. + """Clear cache statistics and invalidate cached entries. - All of them on a function without parameters too, where invalidate_cache() with no arguments - deletes only the entry of the call without arguments, though a key= that reads its context can - file that call under many keys. + An L1-only function loses every entry, in every context: on a function without parameters, a + key= that reads its context can file the one call under many keys. A backed function loses what + invalidate_cache() with no arguments deletes: every entry when it takes parameters, and without + parameters the calling context's key and its pre-0.20.0 twin, in L2 too. """ _stats.clear() if _zero_config_pending: _settle_zero_config() - # In L1-only mode, invalidation is synchronous (no backend I/O needed) - # so cache_clear() works for both sync and async functions. - if inspect.iscoroutinefunction(func) and not _l1_only_mode: + # In L1-only mode, invalidation is synchronous (no backend I/O needed), so cache_clear() works + # for both sync and async functions. + if _l1_only_mode: + _delete_then_forget(None) + return + if inspect.iscoroutinefunction(func): raise TypeError( "cache_clear() cannot clear cache for async functions with a backend. Use 'await fn.ainvalidate_cache()' instead." ) - _ready_to_invalidate() - _delete_then_forget(None) + invalidate_cache() # Other processes' invalidations reach this function's L1 through the process's listener. Only a # backed wrapper with an L1 registers: in L1-only mode nothing is shared, so nothing is announced. diff --git a/tests/unit/test_invalidate_no_args.py b/tests/unit/test_invalidate_no_args.py index 4c4504fe..f402226c 100644 --- a/tests/unit/test_invalidate_no_args.py +++ b/tests/unit/test_invalidate_no_args.py @@ -190,6 +190,24 @@ def expensive(query: str) -> str: f"Expected 4 calls after invalidation, got {call_count}. L2 entries survived invalidate_cache() with no args." ) + def test_cache_clear_without_parameters_deletes_the_key_in_l2(self, tmp_path): + """A backed function without parameters: cache_clear() deletes the call's key in L2, as + invalidate_cache() does, even from a decoration that never filled or read it, so recorded + no key to drain.""" + calls = 0 + backend = FileBackend(FileBackendConfig(cache_dir=str(tmp_path), max_size_mb=256)) + + def report() -> int: + nonlocal calls + calls += 1 + return calls + + writer = cache(backend=backend, ttl=300, namespace="test_clear_no_params", l1_enabled=False)(report) + clearer = cache(backend=backend, ttl=300, namespace="test_clear_no_params", l1_enabled=False)(report) + assert (writer(), writer()) == (1, 1) # the second read is served from L2 + clearer.cache_clear() + assert writer() == 2 + def test_file_backend_partial_failure_retains_keys(self, tmp_path): """If L2 delete fails, the key stays in _cached_keys for retry.""" from unittest.mock import patch diff --git a/tests/unit/test_l1_lock_free_hit.py b/tests/unit/test_l1_lock_free_hit.py index 1b619654..10844089 100644 --- a/tests/unit/test_l1_lock_free_hit.py +++ b/tests/unit/test_l1_lock_free_hit.py @@ -202,9 +202,10 @@ def fn(x: int) -> _Value: assert fn(1).n == 1 def test_cache_clear_removes_every_entry_of_a_function_without_parameters(self) -> None: - """On a function without parameters, invalidate_cache() with no arguments deletes the entry of - the call without arguments, but a key= that reads its context files that call under many keys: - cache_clear() removes every one, not only the clearing context's.""" + """On an L1-only function without parameters, invalidate_cache() with no arguments deletes the + entry of the call without arguments, but a key= that reads its context files that call under many + keys: cache_clear() removes every one, not only the clearing context's. (A backed function keeps + invalidate_cache()'s meaning; see test_invalidate_no_args.py.)""" tenant: contextvars.ContextVar[str] = contextvars.ContextVar("tenant") calls: list[str] = [] From 0a96f66d13caaf112fe931601142de81c6a3096e Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 01:32:24 +1100 Subject: [PATCH 04/14] fix(rust): answer the L1-only hit in safe Rust The lock-free hit reached CPython through unsafe code: a hand-made METH_FASTCALL builtin, direct calls to the name descriptors' getters, unchecked casts and a raw pointer for the tests' clock. cachekit-rs holds no unsafe code (SECURITY.md), and the post-merge cargo-geiger check failed on main with six unsafe functions. The hit now goes through PyO3's checked API: a #[pymethods] get, attribute reads for the function's names, cast_exact for memo slots, exact-type checks against PyO3's type objects, and a mutex with an atomic flag for the tests' clock override. cargo-geiger counts no unsafe function, expression or impl in the crate. The crate root forbids unsafe code, so a pull request's build fails on any instead of the post-merge check. The checked calls cost the hit about 280 instructions, most of it the type lookup of the two name reads: l1_hit moves from 2,482 to 2,758 on CPython 3.14 and from 2,594 to 2,935 on 3.12, and the budgets are raised to match. --- SECURITY.md | 2 +- docs/performance.md | 8 +- rust/src/l1_hit.rs | 412 ++++++++-------------------- rust/src/lib.rs | 9 +- tests/performance/ir_baselines.json | 8 +- 5 files changed, 130 insertions(+), 309 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 17a5821f..e7fd647c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -309,7 +309,7 @@ On the tenant-scoped Redis backend, every successful invalidation is announced o | Tier | Timing | Trigger | Checks | |:-----|:------:|:--------|:-------| -| **Fast** | < 3 min | Every PR | cargo-audit, cargo-deny, clippy, machete, pip-audit | +| **Fast** | < 3 min | Every PR | cargo-audit, cargo-deny, clippy, machete, pip-audit. cachekit-rs declares `#![forbid(unsafe_code)]`, so any build of a PR that adds unsafe code to it fails | | **Medium** | < 15 min | Post-merge | cargo-geiger (zero unsafe functions in cachekit-rs). The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | | **Deep** | < 2 hr | Nightly | Sanitizers (ASan, TSan, MSan), security report | diff --git a/docs/performance.md b/docs/performance.md index 04aa348b..f0b273dd 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -10,7 +10,7 @@ > [!TIP] > **Key numbers** (mean of five run medians; GitHub-hosted `ubuntu-latest` runner, AMD EPYC 7763, CPython 3.14.7, 2026-10-31; see [GitHub-Hosted Runner](#github-hosted-runner)): -> - **Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as plain MessagePack)**: 0.311 ± 0.000μs (about 2,500 instructions per call on CPython 3.14, see [Instruction Budgets](#instruction-budgets)) +> - **Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as plain MessagePack)**: 0.311 ± 0.000μs (about 2,760 instructions per call on CPython 3.14, see [Instruction Budgets](#instruction-budgets)) > - **Same hit, 10K-row DataFrame**: 0.31 ± 0.00μs. An L1-only hit never serializes, so payload size barely matters > - **Decorator + L1 hit with an L2 backend configured, same dict**: 284 ± 2μs. With a backend, L1 holds bytes and every hit deserializes them. This dict compresses, so L1 holds its envelope. Measured before hits stopped re-checking bytes they had already checked, which took about 20% off this hit; see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload) > - **Redis L2 hit**, L1 disabled, same dict, Redis in a container on the same runner: 539 ± 3μs @@ -93,7 +93,7 @@ Only an L1-only function keeps a memo. A function with a backend builds its key With ten threads calling one function on the same key at once, `test_concurrent_cache_access` times each call once its thread is running: execution time after scheduling, not request latency. Each sample starts inside its thread's loop, so on a GIL build, where only one thread runs Python at a time, the time a thread waits for its turn falls outside every sample. Its last figures predate the memo, and the hosted workflow does not run it. No guard measures how long a request waits under contention. -The deterministic figure is the `l1_hit` row of the [instruction budgets](#instruction-budgets): about **2,500 instructions per call** on CPython 3.14 and 2,600 on 3.12, against 10,900 and 11,600 when the hit took the object cache's lock, and about 60,000 before the memo and the fast path. +The deterministic figure is the `l1_hit` row of the [instruction budgets](#instruction-budgets): about **2,760 instructions per call** on CPython 3.14 and 2,940 on 3.12, against 10,900 and 11,600 when the hit took the object cache's lock, and about 60,000 before the memo and the fast path. The extension holds no `unsafe` code, so the hit reads the function's names as any attribute access does: those two reads take about 280 of its instructions on 3.14. The raw L1 byte-cache lookup, without the decorator, takes 354–362ns. An L1-only hit does not use that cache. @@ -298,8 +298,8 @@ The regression gate is the instruction budget: `make perf-ir` on your machine, a | Path | What one call does | CPython 3.12 | CPython 3.14 | |------|--------------------|-------------:|-------------:| -| `l1_hit` | `@cache(backend=None)` L1 hit | 2,594 | 2,482 | -| `minimal_l1_hit` | `@cache.minimal(backend=None)` L1 hit | 2,594 | 2,479 | +| `l1_hit` | `@cache(backend=None)` L1 hit | 2,935 | 2,758 | +| `minimal_l1_hit` | `@cache.minimal(backend=None)` L1 hit | 2,936 | 2,758 | | `l2_hit` | `@cache`, L1 disabled, L2 hit, as one in-process single-flight trip | 171,988 | 167,408 | | `miss` | `@cache`, L1 disabled, L2 miss, compute, L2 write, all as one in-process single-flight trip | 211,434 | 203,521 | | `secure_l1_hit` | `@cache.secure` L1 hit (builds its key, decrypts the ciphertext L1 holds) | 139,943 | 140,926 | diff --git a/rust/src/l1_hit.rs b/rust/src/l1_hit.rs index ee2ef66c..8edb38c5 100644 --- a/rust/src/l1_hit.rs +++ b/rust/src/l1_hit.rs @@ -19,19 +19,19 @@ //! therefore kept only while their slot is: at most [`MEMO_MAX_ENTRIES`] keys per function. //! //! Deadlines are read on one clock, [`now`] (`monotonic` in Python), which `ObjectCache` uses too. +//! +//! Like the rest of this crate, the module is safe Rust: every object it touches goes through PyO3's +//! checked API. -use std::ptr; -use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Mutex, MutexGuard, OnceLock, PoisonError}; use std::time::Instant; -use pyo3::exceptions::PyTypeError; -use pyo3::ffi; use pyo3::gc::PyVisit; use pyo3::intern; use pyo3::prelude::*; -use pyo3::types::{PyDict, PyString, PyTuple, PyType}; -use pyo3::PyTraverseError; +use pyo3::types::{PyBool, PyBytes, PyDict, PyFloat, PyInt, PyString, PyTuple}; +use pyo3::{PyTraverseError, PyTypeInfo}; use crate::hit_counter::PyHitCounter; @@ -41,7 +41,7 @@ use crate::hit_counter::PyHitCounter; /// between them, as the memo keeps what it is keyed by. That bounds an entry at a few KiB (a str of /// 4-byte characters also keeps its UTF-8 copy once the key generator encodes it), and it bounds the /// hashing a lookup repeats for strings that are new to each call. -pub(crate) const MEMO_MAX_LEN: ffi::Py_ssize_t = 256; +pub(crate) const MEMO_MAX_LEN: usize = 256; /// Arguments per call, positional and keyword together. pub(crate) const MEMO_MAX_ARGS: usize = 8; /// Memo slots per decorated function. A full memo is emptied rather than evicted slot by slot, so a @@ -55,10 +55,18 @@ pub(crate) const MEMO_MAX_ENTRIES: usize = 1024; /// inherits the zero with the rest of its memory, so readings stay comparable across a fork. static EPOCH: OnceLock = OnceLock::new(); -/// A test's replacement clock, a Python callable returning seconds, or null for the real one. -/// Installed only by tests, through `_set_clock`. A replaced callable is never released, so a -/// reader that loaded the pointer before a replacement still holds a live object. -static CLOCK_OVERRIDE: AtomicPtr = AtomicPtr::new(ptr::null_mut()); +/// A test's replacement clock, a Python callable returning seconds, or None for the real one. +/// Installed only by tests, through `_set_clock`. Held only to copy or swap the reference, never +/// while Python code runs. +static CLOCK_OVERRIDE: Mutex>> = Mutex::new(None); + +/// Whether a replacement clock is installed, so a reading of the real clock takes no lock. +static CLOCK_OVERRIDDEN: AtomicBool = AtomicBool::new(false); + +/// Seconds since the clock's zero. +fn real_now() -> f64 { + EPOCH.get_or_init(Instant::now).elapsed().as_secs_f64() +} /// Seconds on the clock `ObjectCache` and the lock-free hit share, from an arbitrary zero. /// @@ -69,40 +77,42 @@ static CLOCK_OVERRIDE: AtomicPtr = AtomicPtr::new(ptr::null_mut() #[pyfunction] #[pyo3(name = "monotonic")] pub fn now(py: Python<'_>) -> PyResult { - let fake = CLOCK_OVERRIDE.load(Ordering::Acquire); - if fake.is_null() { - return Ok(EPOCH.get_or_init(Instant::now).elapsed().as_secs_f64()); + if !CLOCK_OVERRIDDEN.load(Ordering::Acquire) { + return Ok(real_now()); + } + let clock = CLOCK_OVERRIDE + .lock() + .unwrap_or_else(PoisonError::into_inner) + .as_ref() + .map(|clock| clock.clone_ref(py)); + // None when a restore landed between the two reads. + match clock { + Some(clock) => clock.call0(py)?.extract(py), + None => Ok(real_now()), } - // SAFETY: a non-null override is a strong reference this module owns and never releases. - unsafe { Bound::from_borrowed_ptr(py, fake) } - .call0()? - .extract() } /// Replace the shared clock with a callable returning seconds, or restore the real one with None. /// /// For tests only: the real clock cannot be moved, and a test of expiry or refresh needs both the -/// cache and the lock-free hit to see the same moved time. Each replaced callable is kept alive for -/// the rest of the process. +/// cache and the lock-free hit to see the same moved time. #[pyfunction] #[pyo3(name = "_set_clock")] pub fn set_clock(clock: Option>) { - let new = clock.map_or(ptr::null_mut(), Bound::into_ptr); - CLOCK_OVERRIDE.store(new, Ordering::Release); + let installed = clock.is_some(); + let replaced = std::mem::replace( + &mut *CLOCK_OVERRIDE + .lock() + .unwrap_or_else(PoisonError::into_inner), + clock.map(Bound::unbind), + ); + CLOCK_OVERRIDDEN.store(installed, Ordering::Release); + // Released after the lock: releasing the old clock can run Python code that reads the clock. + drop(replaced); } // ── The memo key ───────────────────────────────────────────────────────────────────────────── -/// The `i`th item of a tuple, borrowed: the item macro where the build has the tuple's layout, a -/// call to `PyTuple_GetItem` on the stable ABI. -/// -/// # Safety -/// `i` must be in bounds. -#[inline] -unsafe fn tuple_item(t: &Bound<'_, PyTuple>, i: usize) -> *mut ffi::PyObject { - t.get_borrowed_item_unchecked(i).as_ptr() -} - /// What an argument spends of its call's [`MEMO_MAX_LEN`], and whether its type must be part of /// the key; None for anything the memo never takes. /// @@ -111,31 +121,26 @@ unsafe fn tuple_item(t: &Bound<'_, PyTuple>, i: usize) -> *mut ffi::PyObject { /// str or None never compares equal to a value of another of those types, so a call of only those /// can be keyed by its argument tuple as it is. A float or bool can (1, 1.0 and True are equal), and /// a bytes compared with an equal str warns under `python -b`, so their types go in the key. -/// -/// # Safety -/// `a` must be a valid object pointer. #[inline] -unsafe fn admit(a: *mut ffi::PyObject) -> Option<(ffi::Py_ssize_t, bool)> { - let t = ffi::Py_TYPE(a); - if t == &raw mut ffi::PyLong_Type || a == ffi::Py_None() { +fn admit(a: &Bound<'_, PyAny>) -> PyResult> { + let py = a.py(); + let t = a.get_type_ptr(); + Ok(if t == PyInt::type_object_raw(py) || a.is_none() { Some((0, false)) - } else if t == &raw mut ffi::PyUnicode_Type { - Some((ffi::PyUnicode_GetLength(a), false)) - } else if t == &raw mut ffi::PyFloat_Type || t == &raw mut ffi::PyBool_Type { + } else if t == PyString::type_object_raw(py) { + Some((a.len()?, false)) + } else if t == PyFloat::type_object_raw(py) || t == PyBool::type_object_raw(py) { Some((0, true)) - } else if t == &raw mut ffi::PyBytes_Type { - Some((ffi::PyBytes_Size(a), true)) + } else if t == PyBytes::type_object_raw(py) { + Some((a.len()?, true)) } else { None - } + }) } /// An object's type, as a new reference. -/// -/// # Safety -/// `a` must be a valid object pointer. -unsafe fn type_of<'py>(py: Python<'py>, a: *mut ffi::PyObject) -> Bound<'py, PyAny> { - Bound::from_borrowed_ptr(py, ffi::Py_TYPE(a).cast()) +fn type_of<'py>(a: &Bound<'py, PyAny>) -> Bound<'py, PyAny> { + a.get_type().into_any() } /// The memo key of a call's arguments, or None when the memo does not take the call. @@ -169,11 +174,10 @@ pub fn memo_key_of<'py>( if n + nk > MEMO_MAX_ARGS { return Ok(None); } - let mut size: ffi::Py_ssize_t = 0; + let mut size = 0; let mut typed = false; - for i in 0..n { - // SAFETY: i < n = len(args), and args holds valid objects. - match unsafe { admit(tuple_item(args, i)) } { + for a in args.iter_borrowed() { + match admit(&a)? { Some((cost, needs_type)) => { size += cost; typed |= needs_type; @@ -187,49 +191,39 @@ pub fn memo_key_of<'py>( } if !typed { if n == 1 { - // SAFETY: n = 1 = len(args). - let lone = unsafe { args.get_borrowed_item_unchecked(0) }; + let lone = args.get_borrowed_item(0)?; if !lone.is_none() { return Ok(Some(lone.to_owned())); } } return Ok(Some(args.clone().into_any())); } - // SAFETY: as above. - unsafe { - if n == 1 { - let a = tuple_item(args, 0); - return Ok(Some( - PyTuple::new(py, [type_of(py, a), Bound::from_borrowed_ptr(py, a)])?.into_any(), - )); - } - if n == 2 { - let (a, b) = (tuple_item(args, 0), tuple_item(args, 1)); - let items = [ - type_of(py, a), - Bound::from_borrowed_ptr(py, a), - type_of(py, b), - Bound::from_borrowed_ptr(py, b), - ]; - return Ok(Some(PyTuple::new(py, items)?.into_any())); - } - let types = PyTuple::new(py, (0..n).map(|i| type_of(py, tuple_item(args, i))))?; - return Ok(Some( - PyTuple::new(py, [args.as_any(), types.as_any()])?.into_any(), - )); + if n == 1 { + let a = args.get_borrowed_item(0)?.to_owned(); + return Ok(Some(PyTuple::new(py, [type_of(&a), a])?.into_any())); + } + if n == 2 { + let a = args.get_borrowed_item(0)?.to_owned(); + let b = args.get_borrowed_item(1)?.to_owned(); + let items = [type_of(&a), a, type_of(&b), b]; + return Ok(Some(PyTuple::new(py, items)?.into_any())); } + let types = PyTuple::new(py, args.iter_borrowed().map(|a| type_of(&a)))?; + return Ok(Some( + PyTuple::new(py, [args.as_any(), types.as_any()])?.into_any(), + )); } let mut items: Vec<(Bound<'py, PyAny>, Bound<'py, PyAny>)> = Vec::with_capacity(nk); for (name, value) in kwargs.iter() { - // SAFETY: both are valid objects. A str subclass name equals its str and so would share the - // slot, but the key generator refuses it, so it must take the full path. - let Some((cost, _)) = (unsafe { admit(value.as_ptr()) }) else { + // A str subclass name equals its str and so would share the slot, but the key generator + // refuses it, so it must take the full path. + let Some((cost, _)) = admit(&value)? else { return Ok(None); }; - if unsafe { ffi::Py_TYPE(name.as_ptr()) } != &raw mut ffi::PyUnicode_Type { + if !name.is_exact_instance_of::() { return Ok(None); } - size += unsafe { ffi::PyUnicode_GetLength(name.as_ptr()) } + cost; + size += name.len()? + cost; items.push((name, value)); } if size > MEMO_MAX_LEN { @@ -246,11 +240,7 @@ pub fn memo_key_of<'py>( if let Some(e) = failed { return Err(e); } - // SAFETY: args holds valid objects, and i < n. - let types = PyTuple::new( - py, - (0..n).map(|i| unsafe { type_of(py, tuple_item(args, i)) }), - )?; + let types = PyTuple::new(py, args.iter_borrowed().map(|a| type_of(&a)))?; let item_types = PyTuple::new(py, items.iter().map(|(_, v)| v.get_type().into_any()))?; let pairs = PyTuple::new( py, @@ -372,8 +362,6 @@ impl HitEntry { #[pyclass(frozen, module = "cachekit._rust_serializer")] pub struct HitMemo { memo: Py, - /// `HitEntry`'s type, held so a hit checks a slot's type with one comparison. - entry_type: Py, counter: Py, func: Py, module: Py, @@ -381,163 +369,57 @@ pub struct HitMemo { /// The attribute names `__module__` and `__qualname__`, held so a hit looks neither up. module_attr: Py, qualname_attr: Py, - /// For a plain Python function, the descriptors its names are read through. - descriptors: Option, -} - -/// The descriptors `PyObject_GetAttr` reads a plain Python function's `__module__` and -/// `__qualname__` through, with their getters. -/// -/// Calling a getter directly is the same read without the type lookup that takes most of the -/// attribute access: both are data descriptors on `types.FunctionType`, an immutable type, so an -/// attribute access on an exact function always resolves to them. -struct NameDescriptors { - module: Py, - module_get: ffi::descrgetfunc, - qualname: Py, - qualname_get: ffi::descrgetfunc, -} - -impl NameDescriptors { - /// The descriptors for `func`, or None when it is not an exact `types.FunctionType` or a lookup - /// fails; the memo then reads its names with `PyObject_GetAttr`, as any attribute access does. - fn of(func: &Bound<'_, PyAny>) -> Option { - let py = func.py(); - let ty = func.get_type(); - let function_type = py.import("types").ok()?.getattr("FunctionType").ok()?; - if !ty.is(&function_type) { - return None; - } - let namespace = ty.getattr("__dict__").ok()?; - let descriptor = |name: &str| -> Option<(Py, ffi::descrgetfunc)> { - let d = namespace.get_item(name).ok()?; - // SAFETY: d's type is a valid type object; PyType_GetSlot accepts static types since 3.10. - let slot = - unsafe { ffi::PyType_GetSlot(d.get_type().as_type_ptr(), ffi::Py_tp_descr_get) }; - if slot.is_null() { - return None; - } - // SAFETY: the tp_descr_get slot holds a descrgetfunc. - Some((d.unbind(), unsafe { - std::mem::transmute::<*mut std::ffi::c_void, ffi::descrgetfunc>(slot) - })) - }; - let (module, module_get) = descriptor("__module__")?; - let (qualname, qualname_get) = descriptor("__qualname__")?; - Some(Self { - module, - module_get, - qualname, - qualname_get, - }) - } -} - -/// Whether `get(descr, obj)` is the object `held` (a read that fails counts as changed). -/// -/// # Safety -/// As [`attr_is`], and `get` must be `descr`'s type's `tp_descr_get`. -#[inline] -unsafe fn got_is( - get: ffi::descrgetfunc, - descr: *mut ffi::PyObject, - obj: *mut ffi::PyObject, - held: *mut ffi::PyObject, -) -> bool { - let value = get(descr, obj, ffi::Py_TYPE(obj).cast()); - if value.is_null() { - ffi::PyErr_Clear(); - return false; - } - let same = value == held; - ffi::Py_DECREF(value); - same -} - -/// Whether `obj.name` is the object `held` (an attribute that cannot be read counts as changed). -/// -/// # Safety -/// The thread must be attached to the interpreter, and the three pointers valid objects. -#[inline] -unsafe fn attr_is( - obj: *mut ffi::PyObject, - name: *mut ffi::PyObject, - held: *mut ffi::PyObject, -) -> bool { - let value = ffi::PyObject_GetAttr(obj, name); - if value.is_null() { - ffi::PyErr_Clear(); - return false; - } - let same = value == held; - ffi::Py_DECREF(value); - same } impl HitMemo { /// Whether the function still has the names the memo was built with (an attribute that cannot /// be read counts as changed). - fn names_hold(&self, _py: Python<'_>) -> bool { - let f = self.func.as_ptr(); - // SAFETY: the token attaches this thread, the memo owns every object passed, and each - // getter is its descriptor's (NameDescriptors::of). - unsafe { - match &self.descriptors { - Some(d) => { - got_is(d.module_get, d.module.as_ptr(), f, self.module.as_ptr()) - && got_is( - d.qualname_get, - d.qualname.as_ptr(), - f, - self.qualname.as_ptr(), - ) - } - None => { - attr_is(f, self.module_attr.as_ptr(), self.module.as_ptr()) - && attr_is(f, self.qualname_attr.as_ptr(), self.qualname.as_ptr()) - } - } - } + /// + /// Two attribute reads, about a tenth of the hit's instructions, most of it the type lookup + /// each read makes. Calling the names' descriptors directly would skip that lookup, but only + /// through unsafe code, which this crate forbids. + fn names_hold(&self, py: Python<'_>) -> bool { + let func = self.func.bind(py); + let holds = |name: &Py, held: &Py| { + func.getattr(name.bind(py)) + .is_ok_and(|value| value.is(held)) + }; + holds(&self.module_attr, &self.module) && holds(&self.qualname_attr, &self.qualname) } - /// The call's cached value, or `miss` when this memo cannot answer for it. + /// The call's cached value, or None when this memo cannot answer for it. /// /// Answers when the call's memo key has a slot, its entry is alive and before its deadline, and /// the function has its names; then it stamps the entry's LRU recency and counts the hit. Any - /// other call, a stale or expired one included, gets `miss` and goes to the cache's lock. + /// other call, a stale or expired one included, gets None and goes to the cache's lock. fn lookup( &self, py: Python<'_>, args: &Bound<'_, PyTuple>, kwargs: &Bound<'_, PyDict>, - miss: Bound<'_, PyAny>, - ) -> PyResult> { + ) -> PyResult>> { let Some(key) = memo_key_of(args, kwargs)? else { - return Ok(miss.unbind()); + return Ok(None); }; let Some(found) = self.memo.bind(py).get_item(&key)? else { - return Ok(miss.unbind()); + return Ok(None); }; // A slot holds a HitEntry unless code planted another object (see HitMemo): a miss. - if !ptr::eq( - unsafe { ffi::Py_TYPE(found.as_ptr()) }, - self.entry_type.as_ptr().cast(), - ) { - return Ok(miss.unbind()); - } - // SAFETY: found's type is exactly HitEntry, which cannot be subclassed. - let entry = unsafe { found.cast_unchecked::() }.get(); + let Ok(entry) = found.cast_exact::() else { + return Ok(None); + }; + let entry = entry.get(); if !entry.alive.load(Ordering::Acquire) { - return Ok(miss.unbind()); + return Ok(None); } let now = now(py)?; // `<`, not a negated `>=`: a NaN on either side is a miss if now < entry.fresh_until && self.names_hold(py) { entry.used_at.store(now.to_bits(), Ordering::Relaxed); self.counter.get().hit(); - return Ok(entry.value.clone_ref(py)); + return Ok(Some(entry.value.clone_ref(py))); } - Ok(miss.unbind()) + Ok(None) } /// Drop every slot and every entry's key list, and kill the entries too when `kill`. A full memo @@ -558,62 +440,6 @@ impl HitMemo { } } -/// [`HitMemo::lookup`] as a builtin function bound to its memo: `HitMemo.get` makes one, and the -/// wrapper calls it on every call. A vectorcall with no argument parsing, as PyO3's method -/// trampoline costs a tenth of the hit. -struct HitDef(ffi::PyMethodDef); - -// SAFETY: CPython only reads a method definition, and this one never changes. -unsafe impl Sync for HitDef {} - -static HIT_DEF: HitDef = HitDef(ffi::PyMethodDef { - ml_name: c"get".as_ptr(), - ml_meth: ffi::PyMethodDefPointer { - PyCFunctionFast: hit, - }, - ml_flags: ffi::METH_FASTCALL, - ml_doc: c"get(args, kwargs, miss, /)\n--\n\nThe call's cached value, or miss when the memo cannot answer for it.".as_ptr(), -}); - -/// The body of the builtin `HitMemo.get` returns: `(args, kwargs, miss)`, positional. -/// -/// # Safety -/// CPython calls it attached to the interpreter, with `slf` the `HitMemo` that `HitMemo.get` bound -/// and `nargs` valid objects at `args`. -unsafe extern "C" fn hit( - slf: *mut ffi::PyObject, - args: *mut *mut ffi::PyObject, - nargs: ffi::Py_ssize_t, -) -> *mut ffi::PyObject { - let py = Python::assume_attached(); - let result = if nargs == 3 { - let arg = |i: usize| Bound::from_borrowed_ptr(py, *args.add(i)); - // SAFETY (memo): HitMemo.get binds this function to a HitMemo, and to nothing else. - let memo = Bound::from_borrowed_ptr(py, slf); - match (arg(0).cast_into::(), arg(1).cast_into::()) { - (Ok(call_args), Ok(call_kwargs)) => { - memo.cast_unchecked::() - .get() - .lookup(py, &call_args, &call_kwargs, arg(2)) - } - _ => Err(PyTypeError::new_err( - "get() takes a tuple, a dict and a miss sentinel", - )), - } - } else { - Err(PyTypeError::new_err(format!( - "get() takes 3 positional arguments ({nargs} given)" - ))) - }; - match result { - Ok(value) => value.into_ptr(), - Err(e) => { - e.restore(py); - ptr::null_mut() - } - } -} - #[pymethods] impl HitMemo { /// A memo for `func`, counting its hits on `counter`, answering while `func` has these names. @@ -625,36 +451,31 @@ impl HitMemo { module: Py, qualname: Py, ) -> Self { - let descriptors = NameDescriptors::of(func.bind(py)); Self { memo: PyDict::new(py).unbind(), - entry_type: py.get_type::().unbind(), counter, func, module, qualname, module_attr: intern!(py, "__module__").clone().unbind(), qualname_attr: intern!(py, "__qualname__").clone().unbind(), - descriptors, } } - /// The lock-free hit, `get(args, kwargs, miss)`: the call's cached value, or `miss` when this - /// memo cannot answer for it (see `lookup`). A builtin bound to this memo; the wrapper keeps one. - #[getter] - fn get<'py>(slf: &Bound<'py, Self>) -> PyResult> { - // SAFETY: HIT_DEF lives for the process and CPython never writes to it; slf is a HitMemo, - // which the new function holds a reference to. - unsafe { - Bound::from_owned_ptr_or_err( - slf.py(), - ffi::PyCFunction_NewEx( - ptr::addr_of!(HIT_DEF.0).cast_mut(), - slf.as_ptr(), - ptr::null_mut(), - ), - ) - } + /// The lock-free hit: the call's cached value, or `miss` when this memo cannot answer for it + /// (see `lookup`). The wrapper keeps the bound method and calls it first on every call. + #[pyo3(signature = (args, kwargs, miss, /))] + fn get( + &self, + py: Python<'_>, + args: &Bound<'_, PyTuple>, + kwargs: &Bound<'_, PyDict>, + miss: &Bound<'_, PyAny>, + ) -> PyResult> { + Ok(match self.lookup(py, args, kwargs)? { + Some(value) => value, + None => miss.clone().unbind(), + }) } /// Point `key`'s slot at `entry`. Call under the cache's lock. @@ -756,17 +577,12 @@ impl HitMemo { fn __traverse__(&self, visit: PyVisit<'_>) -> Result<(), PyTraverseError> { visit.call(&self.memo)?; - visit.call(&self.entry_type)?; visit.call(&self.counter)?; visit.call(&self.func)?; visit.call(&self.module)?; visit.call(&self.qualname)?; visit.call(&self.module_attr)?; visit.call(&self.qualname_attr)?; - if let Some(d) = &self.descriptors { - visit.call(&d.module)?; - visit.call(&d.qualname)?; - } Ok(()) } } diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 83c6525f..a74fce65 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -3,6 +3,10 @@ //! This crate provides thin Python wrappers around the cachekit-core library. //! Business logic lives in cachekit-core, including the untrusted msgpack decode bound //! (`cachekit_core::check_msgpack_structure`). +//! +//! The crate is safe Rust: unsafe code belongs in cachekit-core, whose CI verifies it, so a build +//! rejects any here rather than leaving it to the post-merge cargo-geiger count (SECURITY.md). +#![forbid(unsafe_code)] // Re-export core types for use in Python bindings pub use cachekit_core::{ByteStorage, StorageEnvelope}; @@ -38,8 +42,9 @@ use pyo3::prelude::*; /// PyO3 checks at run time (a concurrent call raises, never races), and in the features enabled here /// cachekit-core's only shared state is its `AtomicU64` nonce counters (its /// `Mutex` handle registry is behind the `ffi` feature, which is off). `l1_hit`'s -/// shared state is atomics (an entry's liveness and recency, the tests' clock -/// override), a `OnceLock` (the clock's zero), one `Mutex` per entry (its memo +/// shared state is atomics (an entry's liveness and recency), a `OnceLock` (the +/// clock's zero), the tests' clock override (a `Mutex` with an atomic flag beside +/// it, held only to copy or swap the reference), one `Mutex` per entry (its memo /// keys, written only under ObjectCache's lock and never held across Python /// code), and each memo's Python dict, whose single operations free-threaded /// CPython runs atomically — no interior mutability the GIL was papering over. PyO3 diff --git a/tests/performance/ir_baselines.json b/tests/performance/ir_baselines.json index 18a191f0..685c7795 100644 --- a/tests/performance/ir_baselines.json +++ b/tests/performance/ir_baselines.json @@ -5,10 +5,10 @@ "byte_storage_store_40b": 2658, "byte_storage_store_64k": 720667, "file_set": 75887, - "l1_hit": 2594, + "l1_hit": 2935, "l2_hit": 171988, "l2_hit_async_metrics": 171370, - "minimal_l1_hit": 2594, + "minimal_l1_hit": 2936, "miss": 211434, "object_cache_put_full": 164514, "secure_l1_hit": 139943, @@ -28,10 +28,10 @@ "byte_storage_store_40b": 2530, "byte_storage_store_64k": 720663, "file_set": 74132, - "l1_hit": 2482, + "l1_hit": 2758, "l2_hit": 167408, "l2_hit_async_metrics": 167447, - "minimal_l1_hit": 2479, + "minimal_l1_hit": 2758, "miss": 203521, "object_cache_put_full": 160511, "secure_l1_hit": 140926, From ea3096ca79b6b1f20613540ca68f1b53134a56a2 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 01:51:11 +1100 Subject: [PATCH 05/14] docs: quote the hosted L1-only hit figure of the safe extension MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Bench run of the previous commit on a GitHub-hosted AMD EPYC 7763 (CPython 3.14.7) timed the 100-user dict's L1-only hit at 0.341 ± 0.001μs and the DataFrame's at 0.34 ± 0.00μs, against 0.311μs and 0.31μs for the hit that reached CPython through unsafe code. The README, the performance guide, the comparison page and the backend pages quote the new figures; the hosted table cites the run for its two L1-only rows. --- README.md | 4 ++-- docs/backends/README.md | 4 ++-- docs/backends/none.md | 2 +- docs/comparison.md | 2 +- docs/performance.md | 20 ++++++++++---------- 5 files changed, 16 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 11556d32..6033d192 100644 --- a/README.md +++ b/README.md @@ -266,8 +266,8 @@ What a call costs inside your process, from the wall-clock guards in `tests/perf | Path | Time per call | |:-----|--------------:| -| `@cache(backend=None)` L1 hit, 100-user nested dict (23.5 KB as MessagePack) | 0.311 ± 0.000 µs | -| `@cache(backend=None)` L1 hit, 10K-row DataFrame | 0.31 ± 0.00 µs | +| `@cache(backend=None)` L1 hit, 100-user nested dict (23.5 KB as MessagePack) | 0.341 ± 0.001 µs | +| `@cache(backend=None)` L1 hit, 10K-row DataFrame | 0.34 ± 0.00 µs | | L1 hit with an L2 backend configured, same dict | 284 ± 2 µs | | Redis L2 hit with L1 disabled, same dict, Redis in a container on the same machine | 539 ± 3 µs | diff --git a/docs/backends/README.md b/docs/backends/README.md index 26f7b3f8..947f717d 100644 --- a/docs/backends/README.md +++ b/docs/backends/README.md @@ -94,7 +94,7 @@ class BaseBackend(Protocol): | Backend | Latency | Persistence | Cross-Process | TTL | Locking | |---------|---------|-------------|---------------|-----|---------| -| **L1 (In-Memory)** | 0.31μs per `backend=None` hit with a scalar argument the memo answers ([measured](../performance.md#github-hosted-runner)); with a backend, a hit also deserializes | No | No | No | No | +| **L1 (In-Memory)** | 0.34μs per `backend=None` hit with a scalar argument the memo answers ([measured](../performance.md#github-hosted-runner)); with a backend, a hit also deserializes | No | No | No | No | | **[File](file.md)** | set = fsync + a scan of every entry, so slow on a large cache; get waits on a concurrent set ([cost model](file.md#performance-characteristics)) | Yes (disk) | Format shared; one writer process | Yes | File locks | | **[Redis](redis.md)** | One network round trip per operation ([not measured here](../performance.md#not-measured-here)) | Yes (RDB/AOF) | Yes | Yes | Yes | | **[Memcached](memcached.md)** | One network round trip per operation ([not measured here](../performance.md#not-measured-here)) | No | Yes | Yes (max 30d) | No | @@ -400,7 +400,7 @@ these still resolve a backend with nothing configured, and get Redis at | Backend | Latency | Use Case | Notes | |---------|---------|----------|-------| -| **L1 (In-Memory)** | 0.31μs per `backend=None` hit with a scalar argument the memo answers ([measured](../performance.md#github-hosted-runner)) | Repeated calls in same process | Process-local only | +| **L1 (In-Memory)** | 0.34μs per `backend=None` hit with a scalar argument the memo answers ([measured](../performance.md#github-hosted-runner)) | Repeated calls in same process | Process-local only | | **File** | set = fsync, flat in cache size, except eviction, rejection and the 30 s rescan, which scan the directory; get waits on a concurrent set | Single-process local caching | Development, scripts, CLI tools | | **Redis** | One network round trip per operation ([not measured here](../performance.md#not-measured-here)) | Shared cache across pods | Recommended production backend | | **CachekitIO** | 42–45ms p50 hit, measured from MEL | Managed SaaS, zero-ops | HTTPS (HTTP/1.1, async calls on a worker thread); depends on vantage, store region and edge hits ([measured](cachekitio.md#characteristics)); closed beta | diff --git a/docs/backends/none.md b/docs/backends/none.md index fbf56773..7eed566c 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -228,7 +228,7 @@ assert calls == ["EUR", "EUR"] ## Characteristics -- Latency: 0.31μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 22 times the instructions; in-memory, no network +- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 22 times the instructions; in-memory, no network - Shared across processes: No (per-process only) - Persistence: No (lost on restart) - TTL support: Yes diff --git a/docs/comparison.md b/docs/comparison.md index cf400ef4..3ed31e12 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -431,7 +431,7 @@ cachekit's own figures, measured on a GitHub-hosted runner (AMD EPYC 7763, CPyth | Operation | cachekit | |-----------|----------| -| Cache hit, `@cache(backend=None)`, 100-user nested dict, one `int` argument (a memoised hit) | 0.311μs | +| Cache hit, `@cache(backend=None)`, 100-user nested dict, one `int` argument (a memoised hit) | 0.341μs | | Cache hit, L1 with an L2 backend configured, same dict (deserializes) | 284μs, measured before hits skipped re-checking bytes already checked (about 20% less since) | | Cache hit, Redis L2 in a container on the same machine, L1 disabled | 539μs | diff --git a/docs/performance.md b/docs/performance.md index f0b273dd..cdec41ed 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -10,8 +10,8 @@ > [!TIP] > **Key numbers** (mean of five run medians; GitHub-hosted `ubuntu-latest` runner, AMD EPYC 7763, CPython 3.14.7, 2026-10-31; see [GitHub-Hosted Runner](#github-hosted-runner)): -> - **Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as plain MessagePack)**: 0.311 ± 0.000μs (about 2,760 instructions per call on CPython 3.14, see [Instruction Budgets](#instruction-budgets)) -> - **Same hit, 10K-row DataFrame**: 0.31 ± 0.00μs. An L1-only hit never serializes, so payload size barely matters +> - **Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as plain MessagePack)**: 0.341 ± 0.001μs (about 2,760 instructions per call on CPython 3.14, see [Instruction Budgets](#instruction-budgets)) +> - **Same hit, 10K-row DataFrame**: 0.34 ± 0.00μs. An L1-only hit never serializes, so payload size barely matters > - **Decorator + L1 hit with an L2 backend configured, same dict**: 284 ± 2μs. With a backend, L1 holds bytes and every hit deserializes them. This dict compresses, so L1 holds its envelope. Measured before hits stopped re-checking bytes they had already checked, which took about 20% off this hit; see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload) > - **Redis L2 hit**, L1 disabled, same dict, Redis in a container on the same runner: 539 ± 3μs > - **Raw L1 byte-cache lookup** (`L1Cache.get`, no decorator): 354–362ns on a shared x86_64 Linux host (see [Measured Figures](#measured-figures)) @@ -44,19 +44,19 @@ uv run pytest tests/performance/test_statistical_rigor.py::test_l1_cache_hit_sta ### GitHub-Hosted Runner -The `Bench` workflow (`.github/workflows/bench.yml`) runs five guards from `tests/performance/` on a GitHub-hosted `ubuntu-latest` runner and writes their run medians into its job summary; run it from the Actions tab to refresh this table. The README quotes it. [Its run on 2026-10-31](https://github.com/cachekit-io/cachekit-py/actions/runs/41178786512) used an AMD EPYC 7763 (4 vCPUs), CPython 3.14.7 and Redis 7 in a container on the same runner. That is a different machine from the table above, so compare figures within one table. +The `Bench` workflow (`.github/workflows/bench.yml`) runs five guards from `tests/performance/` on a GitHub-hosted `ubuntu-latest` runner and writes their run medians into its job summary; run it from the Actions tab to refresh this table. The README quotes it. [Its run on 2026-10-31](https://github.com/cachekit-io/cachekit-py/actions/runs/41178786512) used an AMD EPYC 7763 (4 vCPUs), CPython 3.14.7 and Redis 7 in a container on the same runner. The two L1-only rows come from [a later run that day](https://github.com/cachekit-io/cachekit-py/actions/runs/41210621694) on the same CPU model and Python, which timed the hit after its last change. That is a different machine from the table above, so compare figures within one table. | Path | Guard (`tests/performance/`) | Run median | |------|------------------------------|-----------:| -| Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as MessagePack) | `test_production_realism.py::test_decorator_overhead_complex_dict` | 0.311 ± 0.000μs | -| Decorator + L1 hit, `@cache(backend=None, serializer="auto")`, 10K-row DataFrame | `test_production_realism.py::test_decorator_overhead_dataframe` | 0.31 ± 0.00μs | +| Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as MessagePack) | `test_production_realism.py::test_decorator_overhead_complex_dict` | 0.341 ± 0.001μs | +| Decorator + L1 hit, `@cache(backend=None, serializer="auto")`, 10K-row DataFrame | `test_production_realism.py::test_decorator_overhead_dataframe` | 0.34 ± 0.00μs | | Decorator + L1 hit with an L2 backend configured (in-process, never reached), same 100-user dict | `test_production_realism.py::test_decorator_overhead_l1_hit_with_backend` | 284 ± 2μs (before the walk skip, see [Key Numbers](#key-numbers)) | | Same hit, plaintext and encrypted arms, interleaved | `test_production_realism.py::test_encryption_overhead` | 286 ± 1μs and 295 ± 1μs (before the walk skip) | | Redis L2 hit, L1 disabled, same 100-user dict | `test_production_realism.py::test_redis_l2_roundtrip` | 539 ± 3μs | The job summary prints each figure in the unit its guard reports: the dict's L1-only hit, the hit with a backend configured and the encryption arms in ns, the DataFrame hit and the Redis L2 hit in μs. This table is in μs, so divide the ns figures by 1,000 when you copy them in. -The dict's L1-only hit timed 311ns in every one of its five runs. The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the walk skip, which plaintext hits take and encrypted hits do not (see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)). That is the first hosted run whose band excludes zero, and [the 2026-10-23 run](https://github.com/cachekit-io/cachekit-py/actions/runs/39987019881) put the difference at 10 ± 21μs, so one run does not pin the cost down (see [Not Measured Here](#not-measured-here)). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in [a run on an EPYC 7763 the same day as the table's](https://github.com/cachekit-io/cachekit-py/actions/runs/41177503539). Hosted runners do not always get the same CPU model: [a run of an earlier commit on an AMD EPYC 9V45 runner](https://github.com/cachekit-io/cachekit-py/actions/runs/39964388582/attempts/2) took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit). +The dict's L1-only hit's five run medians agree within 0.6ns either side of their mean (95% band). The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the walk skip, which plaintext hits take and encrypted hits do not (see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)). That is the first hosted run whose band excludes zero, and [the 2026-10-23 run](https://github.com/cachekit-io/cachekit-py/actions/runs/39987019881) put the difference at 10 ± 21μs, so one run does not pin the cost down (see [Not Measured Here](#not-measured-here)). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in [a run on an EPYC 7763 the same day as the table's](https://github.com/cachekit-io/cachekit-py/actions/runs/41177503539). Hosted runners do not always get the same CPU model: [a run of an earlier commit on an AMD EPYC 9V45 runner](https://github.com/cachekit-io/cachekit-py/actions/runs/39964388582/attempts/2) took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit). The workflow runs these guards with this command, against a Redis it starts on the runner: @@ -69,7 +69,7 @@ Without `REDIS_URL`, the Redis guard uses the pytest-redis server the suite star ## Why an L1-Only Hit Costs the Same for Any Payload -With `backend=None`, cachekit keeps the returned object itself in memory and hands that same object back on a hit (see [backend=None](backends/none.md)). Nothing is serialized or deserialized, so the 100-user dict and a 10K-row DataFrame cost about the same: 0.311μs and 0.31μs on the hosted EPYC 7763, for the wrapper's call and one lookup in the extension (see [Decorator + L1 Hit](#decorator--l1-hit-hot-path)). +With `backend=None`, cachekit keeps the returned object itself in memory and hands that same object back on a hit (see [backend=None](backends/none.md)). Nothing is serialized or deserialized, so the 100-user dict and a 10K-row DataFrame cost about the same: 0.341μs and 0.34μs on the hosted EPYC 7763, for the wrapper's call and one lookup in the extension (see [Decorator + L1 Hit](#decorator--l1-hit-hot-path)). With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the walk skip described below, about 910 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip. @@ -77,7 +77,7 @@ For a plaintext cache with the default serializer and integrity checking on, an ### Decorator + L1 Hit (Hot Path) -An L1-only hit takes 0.311μs for the 100-user dict and 0.31μs for the DataFrame on the hosted EPYC 7763 ([GitHub-Hosted Runner](#github-hosted-runner)). The wrapper makes one call into the extension (`HitMemo.get`), which takes no lock and runs: +An L1-only hit takes 0.341μs for the 100-user dict and 0.34μs for the DataFrame on the hosted EPYC 7763 ([GitHub-Hosted Runner](#github-hosted-runner)). The wrapper makes one call into the extension (`HitMemo.get`), which takes no lock and runs: - **The memo key.** The call's arguments become a key: one `int` or `str` argument is its own key, and any other call's key holds its argument tuple. The memo takes a call only when it has at most 8 arguments, every one an `int`, `float`, `bool`, `None`, `str` or `bytes` matched by its exact type, and its strings, bytes and keyword names hold at most 256 characters between them. A `float`, `bool` or `bytes` puts the argument types in the key, so `1`, `1.0` and `True`, which compare equal, keep their own slots. Any other call (a list, a dict, a `datetime`, an `IntEnum`, longer strings) builds its cache key every time and looks it up under the object cache's lock: a hit with one list argument costs about 22 times the instructions of a memoised one. The cache keys themselves are unchanged. - **One dict lookup** in the function's memo, for the entry the object cache stored for that key. @@ -147,7 +147,7 @@ def expensive_function(user_id: int): **L1 gives you** (hosted EPYC 7763 figures, [GitHub-Hosted Runner](#github-hosted-runner)): - No network round trip on a hit -- An L1-only hit in 0.311μs, against 539μs for a Redis L2 hit of the same 100-user dict +- An L1-only hit in 0.341μs, against 539μs for a Redis L2 hit of the same 100-user dict - With an L2 backend configured, an L1 hit of the 100-user dict in 284μs, measured before hits stopped re-checking bytes they had already checked (about 20% less since): it skips the round trip but still deserializes ### 2. Choose the Right Serializer @@ -226,7 +226,7 @@ def get_historical_data(symbol: str, date: str): ### Bottleneck 2: Decorator Overhead -**Problem:** every decorated call pays for its cache key and the decorator's bookkeeping: 0.311μs per L1-only hit on the hosted EPYC 7763 when the memo answers it, and more when the memo does not take the call, so its key is built every time. +**Problem:** every decorated call pays for its cache key and the decorator's bookkeeping: 0.341μs per L1-only hit on the hosted EPYC 7763 when the memo answers it, and more when the memo does not take the call, so its key is built every time. **Mitigations:** - **This is acceptable** for any function slow enough to be worth caching From 1e3fdf776ea990c6dad1fba56e9971ce197d806c Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 14:20:36 +1100 Subject: [PATCH 06/14] docs: correct the figures and wording derived from the old L1-only hit The hosted L1-only hit moved from 0.311 to 0.341us when it became safe Rust, but ratios worked out from the old figure stayed: a backed L1 hit of the 100-user dict is about 830 times the L1-only hit (was 910), a Redis L2 hit about 1,580 times (was 1,730), and a hit with one list argument about 19 times its instructions (53,153 against 2,765 on CPython 3.14; was 22). The guide states the safe hit's cost on both interpreters (about 280 more instructions on 3.14, 340 on 3.12) instead of saying the extension holds no unsafe code, and the README and the hosted-runner section say which run the two L1-only rows come from. An L1-only cache is any plaintext, non-interop decorator that finds no backend configured, not any such decorator, and the memo test's docstring says names are read by attribute access. --- README.md | 4 ++-- docs/api-reference.md | 2 +- docs/backends/none.md | 4 ++-- docs/performance.md | 14 +++++++------- tests/unit/test_key_memo.py | 4 ++-- 5 files changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index d5c57b24..b3a405bf 100644 --- a/README.md +++ b/README.md @@ -264,7 +264,7 @@ def test_cached_function(): ## Performance -What a call costs inside your process, from the wall-clock guards in `tests/performance/` run on a GitHub-hosted `ubuntu-latest` runner (AMD EPYC 7763, 4 vCPUs) with CPython 3.14.7. Each figure is the mean of five run medians with its 95% band. +What a call costs inside your process, from the wall-clock guards in `tests/performance/` run on a GitHub-hosted `ubuntu-latest` runner (AMD EPYC 7763, 4 vCPUs) with CPython 3.14.7. Each figure is the mean of five run medians with its 95% band. The two L1-only rows come from [a later run](https://github.com/cachekit-io/cachekit-py/actions/runs/41210621694) on the same CPU model and Python than the other two. | Path | Time per call | |:-----|--------------:| @@ -274,7 +274,7 @@ What a call costs inside your process, from the wall-clock guards in `tests/perf | Redis L2 hit with L1 disabled, same dict, Redis in a container on the same machine | 539 ± 3 µs | - With no backend, L1 keeps the returned object itself, whatever the payload's size, and a repeat call with a few short scalar arguments is answered by one call into the Rust extension that takes no lock ([which calls](https://github.com/cachekit-io/cachekit-py/blob/main/docs/performance.md#decorator--l1-hit-hot-path)). Any other call builds its key every time, as does every call to a function with a backend. -- With an L2 backend, L1 holds the serialized bytes and every hit deserializes them: for this dict, about 910 times the L1-only hit. This run predates the Rust decode every such hit now runs, which bounds and decodes the MessagePack in one pass at about half the instructions. Encryption adds a decrypt to each such hit; this run measured the difference at 9.3 ± 1.0 µs. +- With an L2 backend, L1 holds the serialized bytes and every hit deserializes them: for this dict, about 830 times the L1-only hit. This run predates the Rust decode every such hit now runs, which bounds and decodes the MessagePack in one pass at about half the instructions. Encryption adds a decrypt to each such hit; this run measured the difference at 9.3 ± 1.0 µs. - An L2 hit adds the backend's round trip. A Redis on another machine adds its network latency to the last row. Hosted runners come with other CPU models too: in a run of an earlier commit, an EPYC 9V45 runner took about half as long on every row, and other machines give other figures. The [Performance Guide](https://github.com/cachekit-io/cachekit-py/blob/main/docs/performance.md#github-hosted-runner) names the guard behind each figure and how to run it, and the repository's `Bench` workflow repeats the run. diff --git a/docs/api-reference.md b/docs/api-reference.md index c635dc26..618d245d 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -82,7 +82,7 @@ def your_function(args): #### Core Parameters -- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`, or any decorator that finds no backend configured, per [Nothing Configured](backends/README.md#4-nothing-configured-l1-only)) times it, and its stale-while-revalidate refresh, on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). +- **`ttl`** (`int | None`, default: `None` on bare `@cache`; each preset sets its own — see the [preset matrix](configuration.md#intent-presets)) - Cache time-to-live in seconds. `None` = no expiration; on a preset that is an explicit opt-in. An L1-only cache (`backend=None`, `@cache.local()`, or any plaintext, non-interop decorator that finds no backend configured, per [Nothing Configured](backends/README.md#4-nothing-configured-l1-only)) times it, and its stale-while-revalidate refresh, on the OS monotonic clock, which a patched `time.monotonic` does not move ([testing expiry](backends/none.md#ttls-follow-the-os-monotonic-clock)). - **`namespace`** (`str | None`, default: `None`) - Cache key prefix for organization - **`serializer`** (`str | SerializerProtocol`, default: `"default"`) - Serializer name (`"default"`, `"std"`, `"auto"`, `"arrow"`, `"orjson"`) or `SerializerProtocol` instance - **`integrity_checking`** (`bool`, default: `True`; any non-bool raises `ConfigurationError`) - Enable xxHash3-64 checksums for corruption detection (non-cryptographic — detects bit rot and storage bugs, NOT tampering; tamper resistance requires encryption). Behaviour when a writer and reader disagree on this flag: [cross-config reads](serializers/auto.md#cross-config-reads-integrity_checking-mismatch) diff --git a/docs/backends/none.md b/docs/backends/none.md index f171d371..4273b11d 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -205,7 +205,7 @@ assert len(calls) == 2 # ran on both calls: nothing was stored ## TTLs Follow the OS Monotonic Clock -An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for any decorator that finds no backend configured, as it runs in this mode ([Nothing Configured](README.md#4-nothing-configured-l1-only)). A cache with a backend is unchanged. +An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for any plaintext, non-interop decorator that finds no backend configured, as it runs in this mode ([Nothing Configured](README.md#4-nothing-configured-l1-only)). A cache with a backend is unchanged. To test what happens after an entry expires, remove it instead: @@ -229,7 +229,7 @@ assert calls == ["EUR", "EUR"] ## Characteristics -- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 22 times the instructions; in-memory, no network +- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 19 times the instructions; in-memory, no network - Shared across processes: No (per-process only) - Persistence: No (lost on restart) - TTL support: Yes diff --git a/docs/performance.md b/docs/performance.md index 50cacc4e..127ccba4 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -44,7 +44,7 @@ uv run pytest tests/performance/test_statistical_rigor.py::test_l1_cache_hit_sta ### GitHub-Hosted Runner -The `Bench` workflow (`.github/workflows/bench.yml`) runs five guards from `tests/performance/` on a GitHub-hosted `ubuntu-latest` runner and writes their run medians into its job summary; run it from the Actions tab to refresh this table. The README quotes it. [Its run on 2026-10-31](https://github.com/cachekit-io/cachekit-py/actions/runs/41178786512) used an AMD EPYC 7763 (4 vCPUs), CPython 3.14.7 and Redis 7 in a container on the same runner. The two L1-only rows come from [a later run that day](https://github.com/cachekit-io/cachekit-py/actions/runs/41210621694) on the same CPU model and Python, which timed the hit after its last change. That is a different machine from the table above, so compare figures within one table. +The `Bench` workflow (`.github/workflows/bench.yml`) runs five guards from `tests/performance/` on a GitHub-hosted `ubuntu-latest` runner and writes their run medians into its job summary; run it from the Actions tab to refresh this table. The README quotes it. [Its run on 2026-10-31](https://github.com/cachekit-io/cachekit-py/actions/runs/41178786512) used an AMD EPYC 7763 (4 vCPUs), CPython 3.14.7 and Redis 7 in a container on the same runner. That is a different machine from the table above, so compare figures within one table. The two L1-only rows come from [a later run that day](https://github.com/cachekit-io/cachekit-py/actions/runs/41210621694) on the same CPU model and Python, which timed the hit after its last change. | Path | Guard (`tests/performance/`) | Run median | |------|------------------------------|-----------:| @@ -56,7 +56,7 @@ The `Bench` workflow (`.github/workflows/bench.yml`) runs five guards from `test The job summary prints each figure in the unit its guard reports: the dict's L1-only hit, the hit with a backend configured and the encryption arms in ns, the DataFrame hit and the Redis L2 hit in μs. This table is in μs, so divide the ns figures by 1,000 when you copy them in. -The dict's L1-only hit's five run medians agree within 0.6ns either side of their mean (95% band). The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the decode changes; since then, a plaintext hit on a kept entry skips envelope decompression and checksum validation, which an encrypted hit does not (see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)). That is the first hosted run whose band excludes zero, and [the 2026-10-23 run](https://github.com/cachekit-io/cachekit-py/actions/runs/39987019881) put the difference at 10 ± 21μs, so one run does not pin the cost down (see [Not Measured Here](#not-measured-here)). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in [a run on an EPYC 7763 the same day as the table's](https://github.com/cachekit-io/cachekit-py/actions/runs/41177503539). Hosted runners do not always get the same CPU model: [a run of an earlier commit on an AMD EPYC 9V45 runner](https://github.com/cachekit-io/cachekit-py/actions/runs/39964388582/attempts/2) took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit). +In the later run, the dict's L1-only hit's five run medians agree within 0.6ns either side of their mean (95% band). The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the decode changes; since then, a plaintext hit on a kept entry skips envelope decompression and checksum validation, which an encrypted hit does not (see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)). That is the first hosted run whose band excludes zero, and [the 2026-10-23 run](https://github.com/cachekit-io/cachekit-py/actions/runs/39987019881) put the difference at 10 ± 21μs, so one run does not pin the cost down (see [Not Measured Here](#not-measured-here)). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in [a run on an EPYC 7763 the same day as the table's](https://github.com/cachekit-io/cachekit-py/actions/runs/41177503539). Hosted runners do not always get the same CPU model: [a run of an earlier commit on an AMD EPYC 9V45 runner](https://github.com/cachekit-io/cachekit-py/actions/runs/39964388582/attempts/2) took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit). The workflow runs these guards with this command, against a Redis it starts on the runner: @@ -71,7 +71,7 @@ Without `REDIS_URL`, the Redis guard uses the pytest-redis server the suite star With `backend=None`, cachekit keeps the returned object itself in memory and hands that same object back on a hit (see [backend=None](backends/none.md)). Nothing is serialized or deserialized, so the 100-user dict and a 10K-row DataFrame cost about the same: 0.341μs and 0.34μs on the hosted EPYC 7763, for the wrapper's call and one lookup in the extension (see [Decorator + L1 Hit](#decorator--l1-hit-hot-path)). -With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the decode changes described below, about 910 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip. +With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the decode changes described below, about 830 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip. Every read with the default serializer decodes its MessagePack in the Rust extension, in one pass that also enforces the bounds on untrusted MessagePack: nesting at most 1,024 levels deep, and no header declaring more elements or bytes than the rest of the input can back. That covers L2 reads, L1 hits and the decrypted plaintext of encrypted reads. The decode builds the Python objects directly. It shares most repeated short strings, map keys up to 64 bytes and values up to 32, as one object within a value (a small table per decode; two strings that land in one slot replace each other). It runs the Python hook that restores datetimes only on maps holding a `__datetime__`, `__date__` or `__time__` key, because the hook hands back any other map unchanged. It returns the same values and refuses the same inputs with the same exception classes as the decode it replaced: cachekit-core's structure walk, then msgpack-python's C unpacker calling the hook once per map. On the 100-user dict (23,547 bytes of MessagePack, about 3,400 objects), the decode fell from 2,580,104 to 1,238,204 instructions on CPython 3.14 (−52.0%, callgrind main-thread instructions per call, the [Instruction Budgets](#instruction-budgets) method), level with `pickle.loads` of the same value (1,203,437). The 1,000-user dict (239 KB) fell 51.0% and a 1,090-byte one 45.3%. A list of 200 records each holding a datetime fell 33.2%, since every datetime still runs the hook. On the abi3t extension that CPython 3.15 and later load, the 100-user dict fell 50.5%. @@ -81,7 +81,7 @@ For a plaintext cache with the default serializer and integrity checking on, an An L1-only hit takes 0.341μs for the 100-user dict and 0.34μs for the DataFrame on the hosted EPYC 7763 ([GitHub-Hosted Runner](#github-hosted-runner)). The wrapper makes one call into the extension (`HitMemo.get`), which takes no lock and runs: -- **The memo key.** The call's arguments become a key: one `int` or `str` argument is its own key, and any other call's key holds its argument tuple. The memo takes a call only when it has at most 8 arguments, every one an `int`, `float`, `bool`, `None`, `str` or `bytes` matched by its exact type, and its strings, bytes and keyword names hold at most 256 characters between them. A `float`, `bool` or `bytes` puts the argument types in the key, so `1`, `1.0` and `True`, which compare equal, keep their own slots. Any other call (a list, a dict, a `datetime`, an `IntEnum`, longer strings) builds its cache key every time and looks it up under the object cache's lock: a hit with one list argument costs about 22 times the instructions of a memoised one. The cache keys themselves are unchanged. +- **The memo key.** The call's arguments become a key: one `int` or `str` argument is its own key, and any other call's key holds its argument tuple. The memo takes a call only when it has at most 8 arguments, every one an `int`, `float`, `bool`, `None`, `str` or `bytes` matched by its exact type, and its strings, bytes and keyword names hold at most 256 characters between them. A `float`, `bool` or `bytes` puts the argument types in the key, so `1`, `1.0` and `True`, which compare equal, keep their own slots. Any other call (a list, a dict, a `datetime`, an `IntEnum`, longer strings) builds its cache key every time and looks it up under the object cache's lock: a hit with one list argument costs about 19 times the instructions of a memoised one. The cache keys themselves are unchanged. - **One dict lookup** in the function's memo, for the entry the object cache stored for that key. - **Its liveness and deadline.** The entry must still be stored, and its deadline not yet reached: its hard expiry, or with SWR the earliest point a refresh could be flagged (`ttl * swr_threshold_ratio * 0.9` after it was stored), past which the hit goes to the lock, which decides on the refresh. - **The function's names.** The memo answers only while the function's `__module__` and `__qualname__` are the objects it was built with, as its cache keys embed them: renaming the function re-keys its calls. @@ -95,7 +95,7 @@ Only an L1-only function keeps a memo. A function with a backend builds its key With ten threads calling one function on the same key at once, `test_concurrent_cache_access` times each call once its thread is running: execution time after scheduling, not request latency. Each sample starts inside its thread's loop, so on a GIL build, where only one thread runs Python at a time, the time a thread waits for its turn falls outside every sample. Its last figures predate the memo, and the hosted workflow does not run it. No guard measures how long a request waits under contention. -The deterministic figure is the `l1_hit` row of the [instruction budgets](#instruction-budgets): about **2,760 instructions per call** on CPython 3.14 and 2,940 on 3.12, against 10,900 and 11,600 when the hit took the object cache's lock, and about 60,000 before the memo and the fast path. The extension holds no `unsafe` code, so the hit reads the function's names as any attribute access does: those two reads take about 280 of its instructions on 3.14. +The deterministic figure is the `l1_hit` row of the [instruction budgets](#instruction-budgets): about **2,760 instructions per call** on CPython 3.14 and 2,940 on 3.12, against 10,900 and 11,600 when the hit took the object cache's lock, and about 60,000 before the memo and the fast path. The hit is safe Rust, so it reads the function's names as any attribute access does: it costs about 280 more instructions on 3.14 and 340 on 3.12 than the hit that called the CPython C API directly, most of it the type lookup in those two reads. The raw L1 byte-cache lookup, without the decorator, takes 354–362ns. An L1-only hit does not use that cache. @@ -117,7 +117,7 @@ The heap costs about 64 bytes per entry on CPython 3.12 and 72 on 3.14, which th ## L2 Backend (Redis) Performance -An L2 hit with L1 disabled, the 100-user dict (23.5KB as MessagePack) and Redis in a container on the same runner takes 539 ± 3μs on the hosted EPYC 7763: the decorator, the round trip and deserialization. That is about 1,730 times an L1-only hit of the same dict on the same runner. A Redis on another machine adds its network latency on top; no guard measures that. +An L2 hit with L1 disabled, the 100-user dict (23.5KB as MessagePack) and Redis in a container on the same runner takes 539 ± 3μs on the hosted EPYC 7763: the decorator, the round trip and deserialization. That is about 1,580 times an L1-only hit of the same dict on the same runner. A Redis on another machine adds its network latency on top; no guard measures that. ## Latency by Serving Layer @@ -236,7 +236,7 @@ def get_historical_data(symbol: str, date: str): ### Bottleneck 1: The L2 Round Trip -**Problem:** a Redis L2 hit takes 539μs on the hosted EPYC 7763, about 1,730 times an L1-only hit of the same 100-user dict (23.5KB as MessagePack). A Redis on another machine adds its network latency. +**Problem:** a Redis L2 hit takes 539μs on the hosted EPYC 7763, about 1,580 times an L1-only hit of the same 100-user dict (23.5KB as MessagePack). A Redis on another machine adds its network latency. **Mitigations:** - **Keep L1 on** (the default), so repeat reads skip the round trip diff --git a/tests/unit/test_key_memo.py b/tests/unit/test_key_memo.py index e086c182..9b4732ab 100644 --- a/tests/unit/test_key_memo.py +++ b/tests/unit/test_key_memo.py @@ -370,8 +370,8 @@ def test_a_rename_after_the_first_call_rekeys(self, is_async: bool) -> None: assert ":elsewhere.renamed:" in moved def test_a_rename_rekeys_a_callable_that_is_not_a_plain_function(self) -> None: - """A plain function's names are read through the descriptors on its type; any other callable, - here a bound method reporting its function's names, through its own attribute access.""" + """The memo reads every callable's names by attribute access, so a rename re-keys any callable, + here a bound method reporting its function's names, as it re-keys a plain function.""" class Service: def lookup(self, x: int) -> int: From c9caf41b8e3612a3398ba1072bb1c335c92ded86 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 04:03:33 +0000 Subject: [PATCH 07/14] fix: satisfy clippy's doc_markdown and state what the geiger check sees The crate-level doc names CPython without backticks, which the pedantic clippy run in Security Fast rejects. The Medium-tier row said "zero unsafe functions in cachekit-rs", which reads as a fact about the crate; with msgpack_decode exempt from the deny, the check fails on its unsafe functions, and the row now says so. --- SECURITY.md | 2 +- rust/src/lib.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 2100eadf..403ff3ac 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -310,7 +310,7 @@ On the tenant-scoped Redis backend, every successful invalidation is announced o | Tier | Timing | Trigger | Checks | |:-----|:------:|:--------|:-------| | **Fast** | < 3 min | Every PR | cargo-audit, cargo-deny, clippy, machete, pip-audit. cachekit-rs declares `#![deny(unsafe_code)]`, so a build of a PR that adds unsafe code to it fails; `msgpack_decode`, which calls the CPython C API directly, is the one module exempt | -| **Medium** | < 15 min | Post-merge | cargo-geiger (zero unsafe functions in cachekit-rs). The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | +| **Medium** | < 15 min | Post-merge | cargo-geiger. It fails if cachekit-rs holds any unsafe function, so `msgpack_decode`'s unsafe functions fail it. The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | | **Deep** | < 2 hr | Nightly | Sanitizers (ASan, TSan, MSan), security report |
diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 5c1da595..587ff963 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -6,7 +6,7 @@ //! //! Unsafe code belongs in cachekit-core, whose CI verifies it, so the crate denies it: a build //! rejects any here rather than leaving it to the post-merge cargo-geiger count (SECURITY.md). -//! `msgpack_decode` is the one module exempt; it calls the CPython C API directly. +//! `msgpack_decode` is the one module exempt; it calls the `CPython` C API directly. #![deny(unsafe_code)] // Re-export core types for use in Python bindings From 421a4dc4fa1e9c6190600ebb52311a78ddcffd12 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 04:58:53 +0000 Subject: [PATCH 08/14] docs: name the check that rejects unsafe code in cachekit-rs Security Fast's clippy builds without the python feature, so it never compiles l1_hit, python_bindings or msgpack_decode, and deny(unsafe_code) gives way to an allow(unsafe_code) added in the same change. The crate doc and SECURITY.md now say what does run: CI's Lint Rust clippy, on default features, rejects unsafe code outside msgpack_decode unless the change also adds an allow, and the post-merge cargo-geiger check counts it either way. The sentence leaves the Fast row, whose workflow does not run that check. The 830 and 1,580 ratios divide by an L1-only hit from a later run on another VM, so they compare figures on the same CPU model, not the same runner. Fix "than the other two" in the README. --- README.md | 2 +- SECURITY.md | 4 +++- docs/performance.md | 4 ++-- rust/src/lib.rs | 8 +++++--- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index b3a405bf..918b43e9 100644 --- a/README.md +++ b/README.md @@ -264,7 +264,7 @@ def test_cached_function(): ## Performance -What a call costs inside your process, from the wall-clock guards in `tests/performance/` run on a GitHub-hosted `ubuntu-latest` runner (AMD EPYC 7763, 4 vCPUs) with CPython 3.14.7. Each figure is the mean of five run medians with its 95% band. The two L1-only rows come from [a later run](https://github.com/cachekit-io/cachekit-py/actions/runs/41210621694) on the same CPU model and Python than the other two. +What a call costs inside your process, from the wall-clock guards in `tests/performance/` run on a GitHub-hosted `ubuntu-latest` runner (AMD EPYC 7763, 4 vCPUs) with CPython 3.14.7. Each figure is the mean of five run medians with its 95% band. The two L1-only rows come from [a later run](https://github.com/cachekit-io/cachekit-py/actions/runs/41210621694) on the same CPU model and Python as the other two. | Path | Time per call | |:-----|--------------:| diff --git a/SECURITY.md b/SECURITY.md index 403ff3ac..478a4d66 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -309,10 +309,12 @@ On the tenant-scoped Redis backend, every successful invalidation is announced o | Tier | Timing | Trigger | Checks | |:-----|:------:|:--------|:-------| -| **Fast** | < 3 min | Every PR | cargo-audit, cargo-deny, clippy, machete, pip-audit. cachekit-rs declares `#![deny(unsafe_code)]`, so a build of a PR that adds unsafe code to it fails; `msgpack_decode`, which calls the CPython C API directly, is the one module exempt | +| **Fast** | < 3 min | Every PR | cargo-audit, cargo-deny, clippy, machete, pip-audit | | **Medium** | < 15 min | Post-merge | cargo-geiger. It fails if cachekit-rs holds any unsafe function, so `msgpack_decode`'s unsafe functions fail it. The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | | **Deep** | < 2 hr | Nightly | Sanitizers (ASan, TSan, MSan), security report | +cachekit-rs declares `#![deny(unsafe_code)]`, and `msgpack_decode`, which calls the CPython C API directly, is the one module exempt. On every PR, the `Lint Rust` clippy step in CI (`.github/workflows/ci.yml`, default features) rejects unsafe code outside `msgpack_decode`, unless the same change adds an `allow(unsafe_code)`. The post-merge cargo-geiger check counts it either way. Fast's clippy builds without the `python` feature, so it never compiles the Python-facing modules. +
📁 Workflow Files diff --git a/docs/performance.md b/docs/performance.md index 127ccba4..ecd78e26 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -71,7 +71,7 @@ Without `REDIS_URL`, the Redis guard uses the pytest-redis server the suite star With `backend=None`, cachekit keeps the returned object itself in memory and hands that same object back on a hit (see [backend=None](backends/none.md)). Nothing is serialized or deserialized, so the 100-user dict and a 10K-row DataFrame cost about the same: 0.341μs and 0.34μs on the hosted EPYC 7763, for the wrapper's call and one lookup in the extension (see [Decorator + L1 Hit](#decorator--l1-hit-hot-path)). -With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the decode changes described below, about 830 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip. +With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same CPU model, measured before the decode changes described below, about 830 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip. Every read with the default serializer decodes its MessagePack in the Rust extension, in one pass that also enforces the bounds on untrusted MessagePack: nesting at most 1,024 levels deep, and no header declaring more elements or bytes than the rest of the input can back. That covers L2 reads, L1 hits and the decrypted plaintext of encrypted reads. The decode builds the Python objects directly. It shares most repeated short strings, map keys up to 64 bytes and values up to 32, as one object within a value (a small table per decode; two strings that land in one slot replace each other). It runs the Python hook that restores datetimes only on maps holding a `__datetime__`, `__date__` or `__time__` key, because the hook hands back any other map unchanged. It returns the same values and refuses the same inputs with the same exception classes as the decode it replaced: cachekit-core's structure walk, then msgpack-python's C unpacker calling the hook once per map. On the 100-user dict (23,547 bytes of MessagePack, about 3,400 objects), the decode fell from 2,580,104 to 1,238,204 instructions on CPython 3.14 (−52.0%, callgrind main-thread instructions per call, the [Instruction Budgets](#instruction-budgets) method), level with `pickle.loads` of the same value (1,203,437). The 1,000-user dict (239 KB) fell 51.0% and a 1,090-byte one 45.3%. A list of 200 records each holding a datetime fell 33.2%, since every datetime still runs the hook. On the abi3t extension that CPython 3.15 and later load, the 100-user dict fell 50.5%. @@ -117,7 +117,7 @@ The heap costs about 64 bytes per entry on CPython 3.12 and 72 on 3.14, which th ## L2 Backend (Redis) Performance -An L2 hit with L1 disabled, the 100-user dict (23.5KB as MessagePack) and Redis in a container on the same runner takes 539 ± 3μs on the hosted EPYC 7763: the decorator, the round trip and deserialization. That is about 1,580 times an L1-only hit of the same dict on the same runner. A Redis on another machine adds its network latency on top; no guard measures that. +An L2 hit with L1 disabled, the 100-user dict (23.5KB as MessagePack) and Redis in a container on the same runner takes 539 ± 3μs on the hosted EPYC 7763: the decorator, the round trip and deserialization. That is about 1,580 times an L1-only hit of the same dict on the same CPU model. A Redis on another machine adds its network latency on top; no guard measures that. ## Latency by Serving Layer diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 587ff963..392ffe9b 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -4,9 +4,11 @@ //! Business logic lives in cachekit-core, including the untrusted msgpack decode bound //! (`cachekit_core::check_msgpack_structure`). //! -//! Unsafe code belongs in cachekit-core, whose CI verifies it, so the crate denies it: a build -//! rejects any here rather than leaving it to the post-merge cargo-geiger count (SECURITY.md). -//! `msgpack_decode` is the one module exempt; it calls the `CPython` C API directly. +//! Unsafe code belongs in cachekit-core, whose CI verifies it, so the crate denies it. CI's +//! `Lint Rust` clippy (default features) rejects unsafe code outside `msgpack_decode` unless the +//! same change adds an `allow(unsafe_code)`; the post-merge cargo-geiger check counts it either +//! way (SECURITY.md). `msgpack_decode` is the one module exempt; it calls the `CPython` C API +//! directly. #![deny(unsafe_code)] // Re-export core types for use in Python bindings From ca5e71717bfa82f93f20fdef01697b0e0cbc998d Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 10:20:21 +0000 Subject: [PATCH 09/14] docs(perf): the 300,000-call paths cost under 3,000 instructions, not about 2,600 With the safe-Rust hit, l1_hit and minimal_l1_hit cost 2,765 on CPython 3.14 and 2,938 on 3.12. --- docs/performance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/performance.md b/docs/performance.md index 8e7db3d3..853eb9fa 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -306,7 +306,7 @@ The regression gate is the instruction budget: `make perf-ir` on your machine, a **Method** (`tests/performance/ir_budget.py`; the measured process is `ir_workload.py`): - Each path runs under `valgrind --tool=callgrind --separate-threads=yes`, and only the main thread is counted. cachekit's background threads (L1 cleanup) vary by tens of percent between identical runs. -- Per-call cost is `(Ir[3000 calls] - Ir[1000 calls]) / 2000`, so interpreter startup (~2 billion instructions) and warmup cancel. Interpreter teardown is skipped. The two real-data paths cost about 60 million instructions per call, so they run 10 and 30 calls after 5 warmup calls; startup still cancels, and two runs agreed within 0.03%. The three paths that cost about 2,600 instructions per call (`l1_hit`, `minimal_l1_hit`, `byte_storage_store_40b`) run 1,000 and 300,000 calls, because startup cancels only to within about a million instructions (next item). +- Per-call cost is `(Ir[3000 calls] - Ir[1000 calls]) / 2000`, so interpreter startup (~2 billion instructions) and warmup cancel. Interpreter teardown is skipped. The two real-data paths cost about 60 million instructions per call, so they run 10 and 30 calls after 5 warmup calls; startup still cancels, and two runs agreed within 0.03%. The three paths that cost under 3,000 instructions per call (`l1_hit`, `minimal_l1_hit`, `byte_storage_store_40b`) run 1,000 and 300,000 calls, because startup cancels only to within about a million instructions (next item). - Startup lands in one of two states. Every measured process imports pandas, which loads pyarrow, and pyarrow's bundled jemalloc starts a background thread. How that thread is scheduled against the main thread decides which state a run starts in, and the two states' totals differ by 0.1 to 1.2 million instructions, by a different amount at each layout and loop size. Each run is a coin flip, on an idle machine as on a loaded one. With the thread turned off (`JE_ARROW_MALLOC_CONF=background_thread:false`) there is one state: 16 runs agreed within 4,800 instructions. A layout whose two runs start in different states is off by up to 350 instructions per call over 2,000 calls. The median of five layouts hides that on most paths, but one layout of a 2,600-instruction path reads up to 13% off, and in 8 of 80 gate runs at 1,000 and 3,000 calls such layouts moved `byte_storage_store_40b`'s figure by 4.2% to 8.4% (2,434 to 2,912 overall). Over 299,000 calls the shift is a few instructions per call: in 20 gate runs each on a loaded machine, its figure stayed within 2,656 to 2,658 (0.08%) on a build with cachekit-core 0.7.2 and 2,690 to 2,696 (0.22%) with 0.7.4, and `l1_hit` and `minimal_l1_hit` did not move in 6 runs each. - The measured process has a fixed environment (`PYTHONHASHSEED=0`, one BLAS/OpenMP thread, a one-day L1 cleanup interval, nothing inherited), seeded log sampling, and main-thread clocks that advance 1μs per read. Code that records its own duration otherwise executes more instructions when it runs slower. The cleanup sweep reads the real clock, so inside a run it would evict entries stamped with the pinned one. - Nothing that runs on real time lands in the measured loop: no background thread wakes, cyclic GC is off, and the GIL switch interval is long enough that the main thread gives up the GIL only where the code releases it. From 0c6174deadd4156238ba7e36a9ae7074c9ba3ce3 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 2 Nov 2026 13:50:44 +1100 Subject: [PATCH 10/14] fix(rust): forbid unsafe code across cachekit-rs again msgpack_decode now holds no unsafe code, so nothing in the crate needs the deny-and-allow scoping. The crate root forbids unsafe code, which no allow or expect in the crate can lift, the decoder drops its own module-level forbid, and SECURITY.md and the module docs say so. cargo-geiger counts no unsafe function, expression or impl in the crate. --- SECURITY.md | 4 ++-- rust/src/l1_hit.rs | 6 +++--- rust/src/lib.rs | 10 +++------- rust/src/msgpack_decode.rs | 4 ---- 4 files changed, 8 insertions(+), 16 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 478a4d66..90b77242 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -310,10 +310,10 @@ On the tenant-scoped Redis backend, every successful invalidation is announced o | Tier | Timing | Trigger | Checks | |:-----|:------:|:--------|:-------| | **Fast** | < 3 min | Every PR | cargo-audit, cargo-deny, clippy, machete, pip-audit | -| **Medium** | < 15 min | Post-merge | cargo-geiger. It fails if cachekit-rs holds any unsafe function, so `msgpack_decode`'s unsafe functions fail it. The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | +| **Medium** | < 15 min | Post-merge | cargo-geiger (zero unsafe functions in cachekit-rs). The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | | **Deep** | < 2 hr | Nightly | Sanitizers (ASan, TSan, MSan), security report | -cachekit-rs declares `#![deny(unsafe_code)]`, and `msgpack_decode`, which calls the CPython C API directly, is the one module exempt. On every PR, the `Lint Rust` clippy step in CI (`.github/workflows/ci.yml`, default features) rejects unsafe code outside `msgpack_decode`, unless the same change adds an `allow(unsafe_code)`. The post-merge cargo-geiger check counts it either way. Fast's clippy builds without the `python` feature, so it never compiles the Python-facing modules. +cachekit-rs declares `#![forbid(unsafe_code)]`, which no `allow(unsafe_code)` or `expect(unsafe_code)` in the crate can lift. On every PR, the `Lint Rust` clippy step in CI (`.github/workflows/ci.yml`, default features) compiles every module and rejects unsafe code anywhere in the crate. Fast's clippy builds without the `python` feature, so it never compiles the Python-facing modules. After merge, the cargo-geiger report counts unsafe code in the crate; its check fails only on unsafe functions.
📁 Workflow Files diff --git a/rust/src/l1_hit.rs b/rust/src/l1_hit.rs index 37c5016e..54b303f3 100644 --- a/rust/src/l1_hit.rs +++ b/rust/src/l1_hit.rs @@ -20,8 +20,8 @@ //! //! Deadlines are read on one clock, [`now`] (`monotonic` in Python), which `ObjectCache` uses too. //! -//! The module is safe Rust, as the crate requires (`deny(unsafe_code)`): every object it touches goes -//! through PyO3's checked API. +//! The module is safe Rust, as the crate requires (`forbid(unsafe_code)`): every object it touches +//! goes through PyO3's checked API. use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Mutex, MutexGuard, OnceLock, PoisonError}; @@ -377,7 +377,7 @@ impl HitMemo { /// /// Two attribute reads, about a tenth of the hit's instructions, most of it the type lookup /// each read makes. Calling the names' descriptors directly would skip that lookup, but only - /// through unsafe code, which the crate denies. + /// through unsafe code, which the crate forbids. fn names_hold(&self, py: Python<'_>) -> bool { let func = self.func.bind(py); let holds = |name: &Py, held: &Py| { diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 392ffe9b..712f5fbc 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -4,12 +4,9 @@ //! Business logic lives in cachekit-core, including the untrusted msgpack decode bound //! (`cachekit_core::check_msgpack_structure`). //! -//! Unsafe code belongs in cachekit-core, whose CI verifies it, so the crate denies it. CI's -//! `Lint Rust` clippy (default features) rejects unsafe code outside `msgpack_decode` unless the -//! same change adds an `allow(unsafe_code)`; the post-merge cargo-geiger check counts it either -//! way (SECURITY.md). `msgpack_decode` is the one module exempt; it calls the `CPython` C API -//! directly. -#![deny(unsafe_code)] +//! Unsafe code belongs in cachekit-core, whose CI verifies it, so the crate forbids it, and no +//! module can allow it back. SECURITY.md says which checks enforce that. +#![forbid(unsafe_code)] // Re-export core types for use in Python bindings pub use cachekit_core::{ByteStorage, StorageEnvelope}; @@ -27,7 +24,6 @@ pub mod python_bindings; // StandardSerializer's one-pass bounded MessagePack decode #[cfg(feature = "python")] -#[allow(unsafe_code)] pub mod msgpack_decode; // The decorator's lock-free L1-hit counter. Not gated: its Python class is, so the counter's diff --git a/rust/src/msgpack_decode.rs b/rust/src/msgpack_decode.rs index 4b77e6a2..e25dc778 100644 --- a/rust/src/msgpack_decode.rs +++ b/rust/src/msgpack_decode.rs @@ -21,10 +21,6 @@ //! of at most `SHORT_LIST` items, a float or an int outside -5..=256 does not: running out of //! memory while building one aborts the process, where msgpack-python raises MemoryError. -// cargo-geiger checks the crate only after a merge; this fails any build with the `python` -// feature (a default one), which every PR's clippy job and wheel builds compile. -#![forbid(unsafe_code)] - use std::convert::Infallible; use cachekit_core::check_msgpack_structure; From f32e2cbdfd0b302304cc6780f1a1f5488563e19b Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 2 Nov 2026 14:07:35 +1100 Subject: [PATCH 11/14] =?UTF-8?q?fix:=20address=20coderabbit=20review=20?= =?UTF-8?q?=E2=80=94=20removing=20an=20L1-only=20entry=20tests=20the=20mis?= =?UTF-8?q?s,=20not=20the=20TTL?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit-Resolved: none.md:210:Distinguish invalidation te --- docs/backends/none.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/backends/none.md b/docs/backends/none.md index 4273b11d..60a8a05c 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -207,7 +207,7 @@ assert len(calls) == 2 # ran on both calls: nothing was stored An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for any plaintext, non-interop decorator that finds no backend configured, as it runs in this mode ([Nothing Configured](README.md#4-nothing-configured-l1-only)). A cache with a backend is unchanged. -To test what happens after an entry expires, remove it instead: +To test what your code does once an entry has expired, remove the entry instead, so the next call misses as it would after expiry. This tests the miss, not the TTL: an entry expires only when the OS clock passes its `ttl`. For example: ```python from cachekit import cache From 4ca65ac2e32510e92c32d44bc60b68ddb4383fff Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 2 Nov 2026 15:13:31 +1100 Subject: [PATCH 12/14] docs(security): say CI's Lint Rust clippy compiles every non-test module Plain cargo clippy without --all-targets does not compile the #[cfg(test)] modules, so the step covers every non-test module, not every module in the crate. --- SECURITY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SECURITY.md b/SECURITY.md index 90b77242..43233d12 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -313,7 +313,7 @@ On the tenant-scoped Redis backend, every successful invalidation is announced o | **Medium** | < 15 min | Post-merge | cargo-geiger (zero unsafe functions in cachekit-rs). The Miri job runs no tests: this crate has none, and byte_storage and encryption are re-exports from cachekit-core | | **Deep** | < 2 hr | Nightly | Sanitizers (ASan, TSan, MSan), security report | -cachekit-rs declares `#![forbid(unsafe_code)]`, which no `allow(unsafe_code)` or `expect(unsafe_code)` in the crate can lift. On every PR, the `Lint Rust` clippy step in CI (`.github/workflows/ci.yml`, default features) compiles every module and rejects unsafe code anywhere in the crate. Fast's clippy builds without the `python` feature, so it never compiles the Python-facing modules. After merge, the cargo-geiger report counts unsafe code in the crate; its check fails only on unsafe functions. +cachekit-rs declares `#![forbid(unsafe_code)]`, which no `allow(unsafe_code)` or `expect(unsafe_code)` in the crate can lift. On every PR, the `Lint Rust` clippy step in CI (`.github/workflows/ci.yml`, default features) compiles every non-test module and rejects unsafe code in them. Fast's clippy builds without the `python` feature, so it never compiles the Python-facing modules. After merge, the cargo-geiger report counts unsafe code in the crate; its check fails only on unsafe functions.
📁 Workflow Files From 777e2b6aec399dd7c88c3d82360f81246fb5b036 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 2 Nov 2026 15:13:32 +1100 Subject: [PATCH 13/14] =?UTF-8?q?fix:=20address=20coderabbit=20review=20?= =?UTF-8?q?=E2=80=94=20name=20the=20TTL=20deadline=20and=20complete=20the?= =?UTF-8?q?=20memo=20comparison?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit-Resolved: none.md:210:Name the TTL deadline expl CodeRabbit-Resolved: none.md:232:Correct the memo performa --- docs/backends/none.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/backends/none.md b/docs/backends/none.md index 60a8a05c..19775f06 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -207,7 +207,7 @@ assert len(calls) == 2 # ran on both calls: nothing was stored An L1-only entry expires, and its stale-while-revalidate refresh and retry back-off come due, on the OS monotonic clock, which the Rust extension reads itself, so the lock-free hit and the cache judge every deadline on one clock. Patching `time.monotonic` (pytest's `monkeypatch`, freezegun's `freeze_time` or `tick`) does not move it, so a test cannot expire an entry or bring a refresh due by moving fake time. The same holds for `@cache.local()`, and for any plaintext, non-interop decorator that finds no backend configured, as it runs in this mode ([Nothing Configured](README.md#4-nothing-configured-l1-only)). A cache with a backend is unchanged. -To test what your code does once an entry has expired, remove the entry instead, so the next call misses as it would after expiry. This tests the miss, not the TTL: an entry expires only when the OS clock passes its `ttl`. For example: +To test what your code does once an entry has expired, remove the entry instead, so the next call misses as it would after expiry. This tests the miss, not the TTL: an entry expires only once its `ttl` has elapsed on the OS clock. For example: ```python from cachekit import cache @@ -229,7 +229,7 @@ assert calls == ["EUR", "EUR"] ## Characteristics -- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 19 times the instructions; in-memory, no network +- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 19 times the instructions of a memoised hit; in-memory, no network - Shared across processes: No (per-process only) - Persistence: No (lost on restart) - TTL support: Yes From 396bab16b39976b7b6c7ff280198e4a86370e113 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 2 Nov 2026 16:34:48 +1100 Subject: [PATCH 14/14] docs(backends): give the memo-miss instruction ratio for the measured case The 19x figure was measured for a hit with one list argument; other unmemoised calls cost different amounts to key, so the L1-only page now says which case it is, as the performance guide does. --- docs/backends/none.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/backends/none.md b/docs/backends/none.md index 19775f06..b2204933 100644 --- a/docs/backends/none.md +++ b/docs/backends/none.md @@ -229,7 +229,7 @@ assert calls == ["EUR", "EUR"] ## Characteristics -- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key, looks it up under the cache's lock and costs about 19 times the instructions of a memoised hit; in-memory, no network +- Latency: 0.34μs per hit with one `int` argument and no TTL, on a GitHub-hosted AMD EPYC 7763 ([measured](../performance.md#github-hosted-runner)), one call into the Rust extension that takes no lock; a call the [memo](../performance.md#decorator--l1-hit-hot-path) does not take builds its key and looks it up under the cache's lock: a hit with one list argument costs about 19 times the instructions of a memoised one; in-memory, no network - Shared across processes: No (per-process only) - Persistence: No (lost on restart) - TTL support: Yes