Skip to content

Preserve fallback duplicate matches while indexing MinHash candidates - #36

Merged
ifuri-validator-agent[bot] merged 1 commit into
mainfrom
ticket/027-fallback-minhash-index
Nov 6, 2026
Merged

ifuri-validator-agent[bot] merged 1 commit into
mainfrom
ticket/027-fallback-minhash-index

Conversation

@tom-sapletta-com

@tom-sapletta-com tom-sapletta-com commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Without datasketch, workspace scans scored every MinHash against every stored signature, making the fallback quadratic. Cache disjoint signature bands to select every candidate that can pass the existing threshold, then preserve the original Jaccard scores, equal-score order and grouping. Rebuild after additions or threshold changes; retain exhaustive comparison with native scoring and unsupported thresholds.

Validation: full locked test suites on Python 3.10 and 3.13 each passed (294 passed, 14 optional-backend skips); affected tests also passed in the existing native-extension environment (63 passed, 10 skips). Ruff and native governance pass. New regressions cover exhaustive parity at floating-point boundaries, additions, threshold changes, native-scorer fallback and group parity. The comparison-budget test fails before this change (1000 comparisons for one match) and passes afterward (1). A bounded sparse-signature benchmark preserved all ordered results while reducing 1,000,000 comparisons to 1,000 (15.35 s to 0.46 s); dense inputs can still require exhaustive scoring.

Ticket: ticket-027

Coordination: PLF-19979 / PLF-19981. Independent protected approval and merge are required.

@ifuri-validator-agent ifuri-validator-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validator approval after policy checks for exact head 7bef05ca07a391b278f7baecf8724c0e4c268718.

Ticket: ticket-027
Correlation ID: tellmesh-ticket-027-20261106
Model: openai/cursor-auto
Reviewed diff chunks: 2
Advisory LLM verdict: APPROVE
Advisory summary: Reviewed all 2 diff chunk(s). The visible implementation uses the pigeonhole property of d+1 disjoint bands to prune fallback MinHash candidates without changing the original Jaccard scoring, threshold comparison, result ordering, or duplicate semantics. It safely falls back to exhaustive comparison for unsupported thresholds and native scoring, and its cache key/invalidation strategy covers additions and threshold changes in the relevant fallback paths. No security defects or unresolved correctness issues are evident in this chunk. | The added regression tests provide strong coverage for fallback candidate selection, exhaustive-score parity, stable ordering, index refresh after additions and threshold changes, native scorer semantics, comparison pruning, and grouping parity. No unresolved correctness or security defects are visible in this chunk.
Advisory findings: none
The LLM output above is advisory and was not used as the approval trust root.
Semantic review prerequisite: not_required; policy 676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7.

Actual PR impact radar

Exact range: adba8c1a82468267e2d1902b8b68ff1fcf35378b...7bef05ca07a391b278f7baecf8724c0e4c268718
Change digest: c81d723e4b48d36037700e28025535da037444a34296ca0967e4c0b9b0630947
Score: 44/100 (M), estimated 50 min, split recommended: true
Affected services/components: repository-wide/unclassified

Machine-readable radar JSONL and SVG
{"actual_change":{"additions":287,"base_sha":"adba8c1a82468267e2d1902b8b68ff1fcf35378b","binary_files":0,"categories":{"code":1,"configuration":1,"docs":2,"tests":1},"change_digest":"c81d723e4b48d36037700e28025535da037444a34296ca0967e4c0b9b0630947","comparison":"adba8c1a82468267e2d1902b8b68ff1fcf35378b...7bef05ca07a391b278f7baecf8724c0e4c268718","deletions":11,"file_count":5,"files":["project/TICKETS.md","project/ticket-027/README.md","project/ticket-027/intent.json","src/redup/core/lsh_matcher.py","tests/test_lsh_matcher.py"],"head_sha":"7bef05ca07a391b278f7baecf8724c0e4c268718","service_count":0,"services":[]},"assessment_mode":"observed-pr","axes":{"coupling":3,"delivery":2,"scope":2,"uncertainty":3,"validation":1},"complexity":"M","confidence":0.9,"diagnostics":["RADAR-ACCEPTANCE-MISSING","RADAR-BUDGET-EXCEEDED"],"estimate":{"budget_minutes":30,"minutes":50,"within_budget":false},"impact":{"components":["project","src/redup","tests"],"files":["project/TICKETS.md","project/ticket-027/README.md","project/ticket-027/intent.json","src/redup/core/lsh_matcher.py","tests/test_lsh_matcher.py"],"public_interfaces":[],"runtime_dependencies":0},"schema":"subactor.ticket-radar/v1","score":44,"split":{"parts":[{"estimated_minutes":14,"name":"Implement project","scope":["project"]},{"estimated_minutes":14,"name":"Implement src/redup","scope":["src/redup"]},{"estimated_minutes":14,"name":"Implement tests","scope":["tests"]},{"estimated_minutes":15,"name":"Validate and project to trackers","scope":["tests","planfile","github/gitlab/jira projections"]}],"reason":"estimated_minutes_exceed_budget","recommended":true},"standards":[{"id":"wellmanifest/dsl","revision":"6c60fc4e0dd1f1bb74f46a7745e28019908d1203","version":"0.1.0-dev"},{"id":"wellmanifest/ticket-lifecycle","revision":"5bf581907a87b46a13a73e6c033d3abe4d9a306f","version":"0.1.0-dev"},{"id":"wellmanifest/git-lifecycle","revision":"7d77d4b7af57e69bc75c3a0290b3a4805c5c4438","version":"0.2.0-dev"},{"id":"wellmanifest/logs","revision":"48c284ef7a069055c0bcb6b900147ce5e65f8b43","version":"0.3.0"}],"ticket_ref":"ticket-027"}
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img"><title>ticket-027: Preserve fallback duplicate matches while indexing MinHash candidates</title><rect width="128" height="128" rx="12" fill="#f8fafc"/><g stroke-width="1"><polygon points="64,55 72,61 69,71 59,71 56,61" fill="none" stroke="#d7dde5"/><polygon points="64,47 80,59 74,78 54,78 48,59" fill="none" stroke="#d7dde5"/><polygon points="64,38 89,56 79,85 49,85 39,56" fill="none" stroke="#d7dde5"/><polygon points="64,30 97,53 84,92 44,92 31,53" fill="none" stroke="#d7dde5"/><polygon points="64,21 105,51 89,99 39,99 23,51" fill="none" stroke="#d7dde5"/><line x1="64" y1="64" x2="64" y2="21" stroke="#aab4c0"/><line x1="64" y1="64" x2="105" y2="51" stroke="#aab4c0"/><line x1="64" y1="64" x2="89" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="39" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="23" y2="51" stroke="#aab4c0"/></g><polygon points="64,47 89,56 79,85 59,71 48,59" fill="#fb923c" fill-opacity="0.45" stroke="#c2410c" stroke-width="2"/><circle cx="64" cy="64" r="3" fill="#c2410c"/><g font-family="sans-serif" font-size="7" fill="#334155"><text x="64" y="11" text-anchor="middle">SCO</text><text x="114" y="48" text-anchor="middle">COU</text><text x="95" y="107" text-anchor="middle">UNC</text><text x="33" y="107" text-anchor="middle">VAL</text><text x="14" y="48" text-anchor="middle">DEL</text></g><text x="64" y="124" text-anchor="middle" font-family="sans-serif" font-size="8" fill="#0f172a">M · 50m</text></svg>
Merge will be attempted after this approval when explicitly authorized. ## Decision record (recomputable)
DECISION D-027-4726
TICKET ticket-027
HEAD_SHA 7bef05ca07a391b278f7baecf8724c0e4c268718
CORRELATION_ID tellmesh-ticket-027-20261106
ACTOR agent:ifuri-validator-agent[bot]
APPLIED_RULE P-CORE-015
INPUT author_login = "tom-sapletta-com"
INPUT observed_checks = ["governance / enforce=PASS","test (3.10)=PASS","governance / remote lifecycle=PASS","test (3.13)=PASS"]
INPUT required_checks = ["test (3.10)","test (3.13)"]
INPUT required_checks_source = "protected registry (env/request)"
INPUT reviewer_login = "ifuri-validator-agent[bot]"
INPUT semantic_review_assessment = {"schema":"subactor.validator/semantic-review-assessment/v1","subject":{"repository":"semcod/redup","pull_request":36,"head_sha":"7bef05ca07a391b278f7baecf8724c0e4c268718","base_sha":"adba8c1a82468267e2d1902b8b68ff1fcf35378b","diff_sha256":"ae5cfb6ca53be7a40e1f47c51c760c17a617cd142c7a05b64bf47c48f77ce6c7"},"policy":{"policy_schema":"subactor.validator/semantic-review-policy/v1","policy_version":1,"policy_sha256":"676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7","required":false,"critical_paths":[],"observed_paths":["project/TICKETS.md","project/ticket-027/README.md","project/ticket-027/intent.json","src/redup/core/lsh_matcher.py","tests/test_lsh_matcher.py"]},"grounding":"full-diff-not-per-finding-proof","execution_authority":false,"status":"not_required","reason":null,"review_sha256":null,"unresolved":[]}
INPUT superseded_checks = []
INPUT ticket_radar_receipt = {"schema":"subactor.ticket-radar/v1","base_sha":"adba8c1a82468267e2d1902b8b68ff1fcf35378b","head_sha":"7bef05ca07a391b278f7baecf8724c0e4c268718","change_digest":"c81d723e4b48d36037700e28025535da037444a34296ca0967e4c0b9b0630947","score":44,"complexity":"M","estimated_minutes":50,"split_recommended":true,"services":[],"authority":"ADVISORY","promotion":"FORBIDDEN"}
VERDICT APPROVE AUTHORITY DETERMINISTIC
REJECTED REQUEST_CHANGES BECAUSE NO_UNSAFE_CHANGE_REASON_FOUND
ADVISORY llm_verdict = "APPROVE" MODEL "openai/cursor-auto"
ASSERT VERDICT_AUTHORITY != "ADVISORY"

@ifuri-validator-agent
ifuri-validator-agent Bot merged commit 41ae4a8 into main Nov 6, 2026
4 checks passed
@ifuri-validator-agent
ifuri-validator-agent Bot deleted the ticket/027-fallback-minhash-index branch November 6, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant