perf(l1): auto and orjson L1 hits skip re-verification and the per-read copy (LAB-12119)

docs/data-flow-architecture.md

@@ -31,7 +31,7 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
 > **Architecture Principles:**
 > - **L1 + L2 Caching**: In-memory bytes cache (L1) + pluggable backend storage (L2) for optimal performance
 > - **Backend Abstraction**: Protocol-based L2 backend layer (Redis, CachekitIO, File, Memcached, custom)
-> - **Bytes-Only Storage**: with a backend configured, L1 stores serialized bytes (encrypted or plaintext), not Python objects; L1-only mode (`backend=None`) stores the object itself. After its first hit, a plaintext entry of the default serializer with integrity checking on is held as its verified MessagePack where that is no longer than the envelope, so later hits skip the decompress and checksum; every hit still runs the bounded decode (see [Storage Architecture](#storage-architecture)); encrypted caches keep the ciphertext
+> - **Bytes-Only Storage**: with a backend configured, L1 stores serialized bytes (encrypted or plaintext), not Python objects; L1-only mode (`backend=None`) stores the object itself. After its first hit, a plaintext entry of the default or auto serializer with integrity checking on is held as its verified MessagePack where that is no longer than the envelope, so later hits skip the decompress and checksum, and an orjson entry as its checksummed JSON, so later hits skip the checksum; every hit still runs the bounded decode (see [Storage Architecture](#storage-architecture)); encrypted caches keep the ciphertext
 > - **Security First**: Decompression bomb protection, checksum validation, size limits, encrypted-at-rest
 > - **Performance Optimized**: Thread affinity, connection pooling, pre-cached serializers
 > - **Flexible Configuration**: L1+L2, L1-only, L2-only, or observability-only modes
@@ -491,10 +491,10 @@ L1-only mode (`backend=None`) skips this layer: it stores the Python object itse
 - **After refactor**: Stores bytes (encrypted or plaintext) → `len(bytes)` for size estimation
 - **Performance improvement**: O(1) size calculation vs recursive traversal
 - **Security improvement**: Encrypted bytes at rest (decrypt-at-read-time only)
-- **Kept entries**: L1 stores, admits and evicts envelopes. The first L1 hit on a plaintext entry of the default serializer with integrity checking on decodes the envelope as every read does: `ByteStorage.retrieve()` checks the envelope and returns the MessagePack its checksum covers, and the Rust decode (`unpackb_standard`) bounds and decodes it in one pass. When the verified MessagePack is no longer than the envelope, L1 then holds it in the envelope's place (`VerifiedMsgpack`, same expiry and LRU place), and later hits skip the decompress and checksum; otherwise L1 keeps the envelope. Every hit still runs the bounded decode. Compressible values are usually longer as MessagePack, so they keep the envelope; small and incompressible ones switch. The swap never grows an entry, so L1 never holds fewer entries than it would hold envelopes, and it never touches an entry stored or removed since the hit read it. Bytes from L2, and bytes still held as bytes, never skip `retrieve()`: L1 marks a kept entry by its type, which no backend returns
+- **Kept entries**: L1 stores, admits and evicts envelopes. The first L1 hit on a plaintext entry of the default serializer with integrity checking on decodes the envelope as every read does: `ByteStorage.retrieve()` checks the envelope and returns the MessagePack its checksum covers, and the Rust decode (`unpackb_standard`) bounds and decodes it in one pass. When the verified MessagePack is no longer than the envelope, L1 then holds it in the envelope's place (`VerifiedPayload`, same expiry and LRU place), and later hits skip the decompress and checksum; otherwise L1 keeps the envelope. The auto serializer does the same for its plain MessagePack envelopes, decoding with its own bounded decode and object hook; its NumPy, Arrow, DataFrame and Series entries keep the envelope and are verified on every hit. The orjson serializer keeps the JSON its xxHash3-64 checksum verified, which is always shorter than its envelope, so later hits skip the checksum. A kept entry records the serializer that verified it, and only that serializer decodes it. Every hit still runs the bounded decode. Compressible values are usually longer as MessagePack, so they keep the envelope; small and incompressible ones switch. The swap never grows an entry, so L1 never holds fewer entries than it would hold envelopes, and it never touches an entry stored or removed since the hit read it. Bytes from L2, and bytes still held as bytes, never skip `retrieve()`: L1 marks a kept entry by its type, which no backend returns
 
 **Hierarchical Read Path:**
-1. Check L1 cache (bytes) → if hit, deserialize and return (the first hit on a plaintext entry of the default serializer with integrity checking on marks it as verified, so later hits on held MessagePack skip the decompress and checksum)
+1. Check L1 cache (bytes) → if hit, deserialize and return (on a plaintext cache with integrity checking on, the first hit keeps the payload it verified where the serializer allows: every orjson entry, and a default or auto MessagePack entry no longer than its envelope, but never an auto NumPy, Arrow, DataFrame or Series entry. Later hits on a kept payload skip the checksum, and for MessagePack the decompress; see Kept entries above)
 2. If L1 miss, check L2 backend (bytes) → if hit, deserialize, populate L1 with bytes, return
 3. If L2 miss, execute function → serialize to bytes → store in L2 and L1 → return
 

docs/features/prometheus-metrics.md

@@ -170,9 +170,11 @@ What each `cache_operation_duration_ms` series times, measured with a monotonic
 | `operation="set"`, a write `write_behind=True` deferred | The backend write alone, recorded when it lands. A write-behind miss that writes on the caller's thread instead is timed as the row above: an async miss under the distributed lock, a sync streaming write ([What stays on the caller's path](write-behind.md#what-stays-on-the-callers-path)), or a value the full writer queue refused ([The writer](write-behind.md#the-writer)) |
 
 `cache_operation_size_bytes` records the length of the bytes a `get` hit read; a `set` records no size. For an L1 `get`, that is the L1 entry:
-on a plaintext cache with the default serializer and integrity checking on, an entry's first L1 hit swaps its envelope for its
-MessagePack when the MessagePack is no longer than the envelope, so later L1 `get`s of it record a size no larger than its L2 `get`. Earlier releases recorded
-the envelope's length on every hit.
+on a plaintext cache with integrity checking on, an entry's first L1 hit can swap its envelope for the payload it verified, so later
+L1 `get`s of it record a size no larger than its L2 `get`. With the default serializer, and with the auto serializer for a plain
+MessagePack entry, the swap happens when the MessagePack is no longer than the envelope; the auto serializer's NumPy, Arrow,
+DataFrame and Series entries keep the envelope and record its length on every hit. With the orjson serializer every entry swaps,
+so later L1 `get`s record its JSON length. Earlier releases recorded the envelope's length on every hit.
 
 > **Changed durations.** Earlier releases recorded every L1 hit as a constant 0.001 ms, and timed
 > `set` from before the L2 read, so each `set` sample also held the miss round trip and the

docs/performance.md

@@ -75,7 +75,7 @@ With an L2 backend configured, L1 holds the serialized bytes instead, and a hit
 
 Every read with the default serializer decodes its MessagePack in the Rust extension, in one pass that also enforces the bounds on untrusted MessagePack: nesting at most 1,024 levels deep, and no header declaring more elements or bytes than the rest of the input can back. That covers L2 reads, L1 hits and the decrypted plaintext of encrypted reads. The decode builds the Python objects directly. It shares most repeated short strings, map keys up to 64 bytes and values up to 32, as one object within a value (a small table per decode; two strings that land in one slot replace each other). It runs the Python hook that restores datetimes only on maps holding a `__datetime__`, `__date__` or `__time__` key, because the hook hands back any other map unchanged. It returns the same values and refuses the same inputs with the same exception classes as the decode it replaced: cachekit-core's structure walk, then msgpack-python's C unpacker calling the hook once per map. On the 100-user dict (23,547 bytes of MessagePack, about 3,400 objects), the decode fell from 2,580,104 to 1,238,204 instructions on CPython 3.14 (−52.0%, callgrind main-thread instructions per call, the [Instruction Budgets](#instruction-budgets) method), level with `pickle.loads` of the same value (1,203,437). The 1,000-user dict (239 KB) fell 51.0% and a 1,090-byte one 45.3%. A list of 200 records each holding a datetime fell 33.2%, since every datetime still runs the hook. On the abi3t extension that CPython 3.15 and later load, the 100-user dict fell 50.5%.
 
-For a plaintext cache with the default serializer and integrity checking on, an entry's first L1 hit decodes it as every read does, and then, when its verified MessagePack is no longer than the envelope, which is true of small values and incompressible ones, L1 holds that MessagePack in the envelope's place. Later hits on it skip the envelope's LZ4 decompress and xxHash3-64 checksum. Every hit still runs the bounded decode, which costs no more than the unbounded one did. The 100-user dict compresses to a 3,171-byte envelope from 23.5KB of MessagePack, so L1 keeps its envelope; no budget gates this large-value hit, and `l1_hit_backed` in [Instruction Budgets](#instruction-budgets) gates the small-value hit on held MessagePack. Encrypted caches always keep the ciphertext envelope, and every hit on one decrypts and decodes.
+For a plaintext cache with the default or auto serializer and integrity checking on, an entry's first L1 hit decodes it as every read does, and then, when its verified MessagePack is no longer than the envelope, which is true of small values and incompressible ones, L1 holds that MessagePack in the envelope's place. Later hits on it skip the envelope's LZ4 decompress and xxHash3-64 checksum. The auto serializer keeps only plain MessagePack envelopes; its NumPy, Arrow, DataFrame and Series entries are verified on every hit. The orjson serializer keeps the JSON under its checksum, which is always shorter than the envelope, so its later hits skip the checksum. Every hit still runs the bounded decode, which costs no more than the unbounded one did. The 100-user dict compresses to a 3,171-byte envelope from 23.5KB of MessagePack, so L1 keeps its envelope; no budget gates this large-value hit, and `l1_hit_backed` and `l1_hit_backed_auto` in [Instruction Budgets](#instruction-budgets) gate the small-value hit on held MessagePack. Encrypted caches always keep the ciphertext envelope, and every hit on one decrypts and decodes.
 
 ### Decorator + L1 Hit (Hot Path)
 
@@ -321,6 +321,7 @@ The regression gate is the instruction budget: `make perf-ir` on your machine, a
 |------|--------------------|-------------:|-------------:|
 | `l1_hit` | `@cache(backend=None)` L1 hit | 2,595 | 2,487 |
 | `l1_hit_backed` | `@cache` with a backend configured, L1 hit (never reaches the in-process dict backend; decodes the verified MessagePack L1 holds after the first hit) | 83,015 | 83,882 |
+| `l1_hit_backed_auto` | `l1_hit_backed` with `serializer="auto"` (decodes the verified MessagePack the auto serializer keeps) | 91,986 | 92,950 |
 | `minimal_l1_hit` | `@cache.minimal(backend=None)` L1 hit | 2,595 | 2,487 |
 | `l2_hit` | `@cache`, L1 disabled, L2 hit, as one in-process single-flight trip | 172,539 | 167,686 |
 | `miss` | `@cache`, L1 disabled, L2 miss, compute, L2 write, all as one in-process single-flight trip | 213,364 | 204,897 |