feat(retrieval): add diversity-aware ranking and duplicate suppression

[Syt3s]

feat(retrieval): add diversity-aware ranking and duplicate suppression

Description

This PR adds opt-in diversity-aware ranking to find and list-mode search. It combines bounded
candidate over-fetching, MMR, source-group limits, logical-resource folding, and semantic
near-duplicate suppression while preserving the existing retrieval path when the option is omitted.

The public request contract is:

{
  "diversity": {
    "strategy": "combined",
    "lambda": 0.7,
    "group_by": "source_root",
    "max_per_group": 2,
    "candidate_multiplier": 4,
    "similarity_threshold": 0.98
  }
}

Human Involvement

• A human participated in the implementation or review loop
• This PR was generated entirely by AI agents without human participation in the loop

The contributor selected the requirement, reviewed the proposed scope, requested implementation and
acceptance, selected the branch, and directed the implementation workflow and documentation format.

Related Issue

Fixes #3942

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Test update

Changes Made

• Added strict shared DiversityOptions validation with public lambda aliasing, bounded candidate
  expansion, and unknown-field rejection.
• Added a standalone diversity selector supporting stable MMR, group caps, normalized abstract hash
  folding, dense cosine near-duplicate folding, and relevance/group fallback when embedding fails.
• Added logical resource and source-root normalization for public resources, explicit user content,
  peer resources, agent skills, and L0/L1/L2 URI variants.
• Added non-empty-only deduplicated_from result serialization and dictionary round trips.
• Added candidate_limit to hierarchical retrieval without changing the default public limit path.
• Applied diversity once after all typed-query candidates are aggregated and capped the final
  candidate pool at 500.
• Added SearchService and HTTP request propagation; list mode accepts diversity while context mode
  rejects it with HTTP 422.
• Added Python Async/Sync SDK, TypeScript SDK, and Rust CLI support with explicit wire-field mapping.
• Added English and Chinese API documentation.
• Added deterministic offline quality/latency fixtures and evaluator output for baseline versus
  combined ranking.

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this on the following platforms:
  • Linux
  • macOS
  • Windows

Commands and results:

Focused Python feature suite:             55 passed
HTTP and candidate-limit supplement:      24 passed
pytest tests/retrieve:                     64 passed
TypeScript Vitest:                         38 passed
TypeScript ESM/CJS/DTS build:              passed
Rust search tests:                         19 passed
Rust diversity payload test:               1 passed
cargo fmt --check --package ov_cli:         passed
Ruff check on changed Python files:         passed
Ruff format check on new Python files:      passed
mypy --follow-imports=skip on new modules:  passed
git diff --check:                           passed

Offline evaluation:

Metric	Baseline	Diversity
Duplicate rate at k	0.50	0.00
Unique source rate at k	0.50	1.00
Recall at k	0.50	1.00
nDCG at k	0.5856	1.00
p95 latency	~0.001 ms	~0.381 ms

The duplicate rate decreased by 100%, source coverage and relevance metrics improved, and measured
fake-embedder p95 overhead remained below 1 ms.

Checklist

• My code follows the project's coding style
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

There are no dependent changes or external runtime dependencies required by this PR.

[Syt3s]

Hi @zhoujh01, this PR adds diversity-aware retrieval to reduce duplicate results and improve source coverage. It has been rebased onto the latest main, and all Python, TypeScript, and Rust checks are passing. Could you please review it when you have a chance? Thank you!