fix(cache): distinguish full input content in cache keys

[linjiaye929]

Summary

The vector disk cache currently uses AbstractSingle.str_repr() as the key. Those strings are also used to estimate scheduling lengths, so distinct requests can share a key:

• Rerank pairs ("ab", "c") and ("a", "bc") both become "abc".
• Images with the same height share a key regardless of pixels or width.
• Audio clips with the same number of samples share a key regardless of sample values.

When an earlier entry wins the cache/inference race, the later request receives the earlier request's result.

This change gives the cache content-aware, input-type-specific SHA-256 keys. Text and rerank inputs use JSON to preserve boundaries and avoid string-aliasing differences. Images include mode, size, pixels, RGBA palette and metadata; audio includes sampling rate, dtype, shape and C-order sample bytes. The resulting identity is serialized with fixed pickle protocol 4 before hashing. No pickle deserialization is added.

Key construction runs in the cache read worker. The same key is reused for the asynchronous write. Scheduling strings and token-usage estimates are unchanged.

Compatibility and cost

• New keys have a content-v1 byte namespace. Legacy string-key entries remain on disk but are not read, so the cache warms again. Existing expiry/eviction handles old entries; this change does not delete them.
• Hashing and serialization are linear in input size and require content copies. No throughput improvement is claimed or benchmarked.
• Identity includes image metadata conservatively. Differences in nested metadata or serialization/library versions can produce extra cache misses. Equivalent native fixtures were checked across fresh processes with different hash seeds.
• Mutable image/audio inputs must remain unchanged until the request finishes, as documented.

Validation

Linux, Python 3.12.14, diskcache 5.6.3, NumPy 1.26.4, Pillow 12.3.0, and CPU-only PyTorch 2.8.0.

From libs/infinity_emb:

python -m pytest tests/unit_test/inference/test_caching_layer.py -q

• 27 passed after the change.
• The same final tests against unmodified 040269b give 15 failed, 12 passed. Failures cover the distinct-input collisions, legacy-key isolation and single key computation.
• Tests use the real diskcache backend, native queue items/futures, and the actual background read/write path. Only inference completion is simulated; these are not real-model inference tests.
• Four adjacent existing debug-engine/async/sync tests passed, with three real-backend cases deselected.
• ruff check . passes. Formatting and codespell pass for the changed files; git diff --check passes.
• mypy infinity_emb reports the same 22 missing-optional-dependency errors on this branch and unmodified main, with identical output. There are no errors in caching_layer.py.
• Full make precommit, downloaded-model, accelerator and server end-to-end suites were not run in this minimal CPU environment.

Checklist

• Read the contribution guidelines.
• Added regression tests and repeated-input cache-hit coverage.
• Documented cache migration and mutable-input behavior.

AI assistance was used to prepare this contribution.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-12T10:37:32.017058Z	52b71ca	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[greptile-apps]

Confidence Score: 4/5

[Medium impact] Changes how cache keys are computed for embeddings.

The PR appears safe to merge, with a non-blocking request for a concurrent lazy-image test.

Findings

1.  Concurrent image reads lack coverage ▶

Summary

Replaces scheduling-string cache keys with input-type-specific SHA-256 keys.

• The vector disk cache matches results to each input’s type and content.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Request input] --> B[Model preprocessing]
  A --> C[Cache read worker]
  C --> D[Build content identity]
  D --> E[Hash namespaced key]
  E --> F{Cache hit?}
  F -->|Yes| G[Complete unfinished request]
  F -->|No| H[Wait for inference result]
  B --> I[Inference result]
  I --> H
  H --> J[Queue write with same key]
  J --> K[Disk cache]

Loading

_{Reviews (1) · Last reviewed commit: "fix(cache): distinguish full input conte..." · Reviewed by Greptile}

[greptile-apps]

Concurrent image reads lack coverage

The new tests finish hashing before simulated inference starts, and their images are already loaded. Production runs both paths concurrently on the same image, including lazy images returned by Image.open. The new image.tobytes() call reads pixels, so these tests cannot catch problems caused by both workers first loading that image. Add a regression test with a lazy image and overlapping cache lookup and preprocessing.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[linjiaye929]

Confirmed with lazy PNG inputs: overlapping first reads could crash Pillow decoding and leave image_embed pending. Fixed in d0d72fa: cached requests decode off-loop before dispatch, with a thread lock that stays held if the awaiter is cancelled. Added native BatchHandler/diskcache overlap tests using the real CLIPImageProcessor; encode_core is an explicit deterministic fixture, with no model download. Direct-image/bytes, repeat-hit, shared-image/cancellation, invalid-pixel and uncached controls pass; 45 focused tests pass locally. AI-assisted follow-up.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52b71caa36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve non-BMP distinctions in text cache keys

For direct Python-engine calls, a non-BMP code point and an explicit UTF-16 surrogate pair are distinct str inputs (for example, "\U0001D11E" and "\uD834\uDD1E"), but default json.dumps serializes both as "\ud834\udd1e". _pre_hash therefore produces the same key for the same input class, so after one input is cached the other can receive its embedding or rerank score instead of being processed independently; use an injective string encoding that does not fold these two representations.

Useful? React with 👍 / 👎.

[linjiaye929]

Confirmed and fixed in d0d72fa. Text keys now use ensure_ascii=False, preserving distinct non-BMP characters and explicit surrogate pairs without losing query/document boundaries or string-alias independence. Added 12 native diskcache cases across embedding, classification and both rerank fields, including lone-surrogate controls and repeat hits. All 45 cache/image tests pass locally. AI-assisted follow-up.