perf(publication): reduce materialization persistence latency

[suguanYang]

Summary

This draft PR optimizes Knowhere publication/materialization persistence for the production-shaped document workload.

Changes

• Add measurement-only publication tracing for sync and async transaction owners.
• Persist chunks and map-unit tokens with PostgreSQL COPY / bulk Core writes.
• Reduce ORM allocation and redundant revision cleanup work.
• Reuse namespace generation locks and remove redundant flush/query round trips.
• Defer map-unit token foreign-key validation to transaction commit.
• Add graph keyword/entity GIN prefiltering while preserving exact Python overlap scoring.
• Reduce namespace snapshot and serving-manifest read amplification.
• Add the global KNOWHERE_PUBLICATION_STRATEGY=baseline|candidate switch.
• Add staged token-index migrations, including the binary token-hash expression index and reversible retirement of superseded lookup indexes.
• Add focused contract coverage and a reproducible publication benchmark harness.

Evidence

On the Mac mini, using a PostgreSQL 15 clone with 170,280 chunks, 107,032 map units, and more than 12 million existing token rows:

• candidate async: 6.017s in the isolated candidate configuration;
• candidate sync: 5.753s in the isolated candidate configuration;
• conservative safe-index candidate: async 9.395s and sync 9.432s on the repeat sample.

The classic retrieval token-selective and frequency queries both use idx_document_map_unit_tokens_token_lookup_binary according to EXPLAIN (ANALYZE, BUFFERS) on the production-shaped Mac clone.

These are production-shaped local measurements, not a production latency guarantee.

Validation

Passed:

• migration contracts: 21 cases;
• publication state/replay/namespace/GIN/COPY/trace SQL contracts: 43 cases;
• Ruff;
• Pyright: 0 errors, 0 warnings;
• git diff --check.

Known local test limitation:

• the classic retrieval contract collection has 14 failures caused by the existing JobStateAuditLog SQLAlchemy mapper registration problem during API lifespan setup. The failures occur before the retrieval assertions execute.

Rollout status

This is intentionally a draft PR.

• No staging or production migration has been applied.
• The default strategy remains baseline.
• Staging is still required for migration, query-plan, consistency, and rollback smoke checks.
• Production capacity and retrieval-interference admission gates are not complete.
• The index-retirement migration must only be applied after the recorded production query-plan/load evidence and rollback checks.

Rollback

Set KNOWHERE_PUBLICATION_STRATEGY=baseline to return application publication behavior to the baseline path. The index migrations have downgrade paths; no production schema mutation is included in this PR.

DevOps rollout item

Tracked separately in Issue #444.

• Add KNOWHERE_PUBLICATION_STRATEGY to API and Worker deployment configuration.
• Keep both services on baseline through migration and readiness checks.
• Verify API and Worker announce the same strategy at startup.
• Apply and verify migrations in staging, including index validity/readiness.
• Verify the binary token expression index in the classic retrieval query plan.
• Run staging retrieval consistency and sync/async publication smoke checks.
• Record production baseline p50/p95, errors, timeouts, WAL, CPU, I/O, and index growth.
• Enable candidate globally only after the operational gates pass.
• Verify application rollback by setting KNOWHERE_PUBLICATION_STRATEGY=baseline and restarting API/Worker.
• Verify schema downgrade only after application rollback, preserving production-used indexes.
• Attach rollout, observation, and rollback evidence before marking this PR ready.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.