diff --git a/AGENT_DIARY.md b/AGENT_DIARY.md index b02f6746..b691a62e 100644 --- a/AGENT_DIARY.md +++ b/AGENT_DIARY.md @@ -124,6 +124,33 @@ - *nit 4–6 исправлены в том же проходе:* `None`/отрицательные счётчики теперь fail-open («неизвестно» ≠ «отсутствует»), мёртвая ветка убрана. - P5 снят и записан до мержа: requests 10→12/25 при планке ≥10. +- **Dense-сборка 37 мин: найдена причина (2026-11-12, `a25627a`)** + - *Симптом:* 7 307 чанков за 2 210.8 s (3.3 ch/s), время не зависит от длины + текста. Батч из 32 коротких текстов стоил столько же (7.9 s), сколько из 32 + длинных (6.2 s). + - *Корень:* `onnx_server.py` настраивал токенизатор на `enable_padding(length=512)` + и оба пути (`_embed_single`, `_embed_batch`) добивали каждый вход до 512; + короткий чанк платил за 512 токенов. Замер прямой сессии: seq=32 → 190.7 ch/s, + seq=512 → 5.1 ch/s (37x). + - *Корпус:* 2.14 M токенов, медиана 287, p99 554; фиксированный padding — + 3.74 M токенов, по-батчевый (сортировка по длине уже есть) — 2.18 M. + - *Fix:* padding до самого длинного текста В БАТЧЕ, truncation 512 сохранён; + `ONNX_FIXED_PADDING=512` возвращает старое поведение для совместимости. + Guard: `tests/test_onnx_dynamic_padding.py` (3 теста). + - *Замер после:* короткий батч 0.06 s (464 ch/s), средний 0.63 s (50 ch/s), + длинный без изменений 6.16 s. Полная пересборка: **7 419 чанков за 1 014.7 s** + (было 2 210.8 s; «до» шло частично под параллельным pytest — сравнение не + чистое, чистый эффект подтверждён коротким батчем и оценкой 2.13x). + - *Корректность:* векторы не бит-в-бит (косинус p50 0.998, p5 0.996, min 0.863 + на 7 059 одинаковых чанках); на одном индексе top-1 совпал 14/15. Полный + quality-panel не гонялся — записано как условие перед включением по умолчанию; + dense `smoke_e2e` PASSED (шаг 4 = vector, onnx, 384). + - *Quality-panel (48 замороженных запросов, тот же hybrid-путь):* control + hit@1/5/MRR = 0.562/0.688/0.601, treatment = 0.583/0.688/0.617; парно + +1/−0, 0/0 (33/48 в обоих), +2/−0; p50 274→258 ms. Ни один запрос не стал + хуже → на этой панели фикс не ухудшает качество (n=48 — не общий вывод). + - *Отчёт:* `experiments/onnx_dense_2026_11/REPORT.md` (+ `quality_panel.py`, + `results_control.json`, `results_treatment.json`). - **После мержей 2026-11-12: main собран, доки проверены, аудит чист** - *Мержи (по указанному порядку):* #116 → `a5135d0`, #115 → `83229e1`, #117 → `183a5f0`, #118 → `8149cb2`, #120 → `64341de`, #121 → `6f60c51`, diff --git a/docs/en/CONFIGURATION.md b/docs/en/CONFIGURATION.md index be5adcde..c49d1969 100644 --- a/docs/en/CONFIGURATION.md +++ b/docs/en/CONFIGURATION.md @@ -92,6 +92,7 @@ path and LSP. | `EMBEDDING_MODEL` / `MODEL_NAME` / `EMBEDDING_DIMENSION` | `qwen3-embedding` / `multilingual-e5-small-int8` / `384` | Provider model selection | | `MSCODEBASE_EMBEDDER_PROVIDER` | `auto` | Pin the embedder (`onnx`, `llama_cpp`, `lm_studio`, `ollama`); external endpoints are only used when named | | `MSCODEBASE_ONNX_PROVIDER` | — | Force a specific ONNX execution provider | +| `ONNX_FIXED_PADDING` | `0` | `512` restores the legacy fixed-width ONNX inputs (old-index parity); `0` pads per batch (faster) | | `MSCODEBASE_REMOTE_TOKEN` / `_REMOTE_RATE_LIMIT_RPS` | — | Remote provider auth and rate limit | | `MSCODEBASE_LANGUAGE_PACK` | `false` | Enable tree-sitter-language-pack (downloads parsers on first use) | diff --git a/docs/ru/CONFIGURATION.md b/docs/ru/CONFIGURATION.md index 4e5d7e71..6d3ec16a 100644 --- a/docs/ru/CONFIGURATION.md +++ b/docs/ru/CONFIGURATION.md @@ -95,6 +95,7 @@ dense-пути и LSP. | `EMBEDDING_MODEL` / `MODEL_NAME` / `EMBEDDING_DIMENSION` | `qwen3-embedding` / `multilingual-e5-small-int8` / `384` | Выбор модели провайдера | | `MSCODEBASE_EMBEDDER_PROVIDER` | `auto` | Закрепить эмбеддер (`onnx`, `llama_cpp`, `lm_studio`, `ollama`); внешние эндпоинты используются только по имени | | `MSCODEBASE_ONNX_PROVIDER` | — | Принудительно задать конкретный execution provider ONNX | +| `ONNX_FIXED_PADDING` | `0` | `512` возвращает прежнюю фиксированную ширину входов ONNX (совместимость со старым индексом); `0` — padding по батчу (быстрее) | | `MSCODEBASE_REMOTE_TOKEN` / `_REMOTE_RATE_LIMIT_RPS` | — | Аутентификация и лимит частоты для удалённого провайдера | | `MSCODEBASE_LANGUAGE_PACK` | `false` | Включить tree-sitter-language-pack (скачивает парсеры при первом использовании) | diff --git a/docs/zh/CONFIGURATION.md b/docs/zh/CONFIGURATION.md index 28fd6074..54cc9970 100644 --- a/docs/zh/CONFIGURATION.md +++ b/docs/zh/CONFIGURATION.md @@ -90,6 +90,7 @@ | `EMBEDDING_MODEL` / `MODEL_NAME` / `EMBEDDING_DIMENSION` | `qwen3-embedding` / `multilingual-e5-small-int8` / `384` | provider 模型选择 | | `MSCODEBASE_EMBEDDER_PROVIDER` | `auto` | 固定 embedder（`onnx`、`llama_cpp`、`lm_studio`、`ollama`）；外部端点仅在显式指定时使用 | | `MSCODEBASE_ONNX_PROVIDER` | — | 强制指定 ONNX 执行 provider | +| `ONNX_FIXED_PADDING` | `0` | `512` 恢复旧的固定宽度 ONNX 输入（旧索引兼容）；`0` 按批次填充（更快） | | `MSCODEBASE_REMOTE_TOKEN` / `_REMOTE_RATE_LIMIT_RPS` | — | 远程 provider 认证与速率限制 | | `MSCODEBASE_LANGUAGE_PACK` | `false` | 启用 tree-sitter-language-pack（首次使用时下载 parser） | diff --git a/experiments/onnx_dense_2026_11/REPORT.md b/experiments/onnx_dense_2026_11/REPORT.md new file mode 100644 index 00000000..9341905d --- /dev/null +++ b/experiments/onnx_dense_2026_11/REPORT.md @@ -0,0 +1,127 @@ +# ONNX dense build: where the 37 minutes went + +- Date: 2026-11-12 +- Base: `main` @ `a25627a` +- Machine: 12 logical CPUs, Windows +- Provider: `onnx:multilingual-e5-small-int8`, output 384-d +- Trigger: the dense build of this repository took **2 210.8 s for 7 307 chunks** + (3.3 chunks/s) and was recorded as "not optimised". + +## Question + +Is the cost in the indexer, the HTTP layer, or the model — and is it fixable +without changing the answer? + +## Measurements + +### 1. Inference scales with sequence length (direct session, batch=32) + +``` +seq= 32 : 0.17 s 190.7 ch/s 6 103 tok/s +seq=128 : 0.86 s 37.1 ch/s 4 747 tok/s +seq=256 : 2.30 s 13.9 ch/s 3 569 tok/s +seq=512 : 6.33 s 5.1 ch/s 2 587 tok/s +``` + +The model is healthy: shorter inputs are up to 37x cheaper. Therefore a build +that costs the same for every chunk is padding every chunk to 512. + +### 2. Before the fix, batch time did not depend on text length + +``` +32 x short (~30 chars) 7.93 s 4.0 ch/s +32 x medium (~300 chars) 6.63 s 4.8 ch/s +32 x long (~2000 chars) 6.21 s 5.2 ch/s +batch 8 / 16 / 32 / 64 4.2 / 5.1 / 5.2 / 4.6 ch/s (no batch-size win) +``` + +Root cause in `src/core/embedder/onnx_server.py`: the tokenizer was configured +with `enable_padding(length=512)` and both `_embed_single` and `_embed_batch` +padded every input to `max_length` (512) before `session.run`. A 31-token chunk +paid for 512 tokens. + +### 3. What the real corpus looks like (7 307 chunks) + +``` +total tokens 2 144 633 +tokens min/median/p90/p99/max = 31 / 287 / 400 / 554 / 971 +fixed 512 padding 3 741 184 padded tokens +dynamic, sorted batches ~2.18 M padded tokens +estimated embedding time: 1 450 s -> 679 s (2.13x) +``` + +The indexer already sorts chunks by length (`index_project_runner.py`, +`_flat_chunks.sort(key=lambda x: len(x[1]))`), so a per-batch maximum is very +close to the true lengths. + +### 4. After the fix (per-batch padding) + +``` +32 x short 0.06 s 464 ch/s +32 x medium 0.63 s 50 ch/s +32 x long 6.16 s 5.2 ch/s (unchanged, as expected) +``` + +Full rebuild of the same repository (fresh data root): + +``` +chunks 7 419 (main gained files since the first build) +wall 1 014.7 s vs 2 210.8 s before (2.18x) +``` + +Caveat: the "before" build ran with a full pytest suite in parallel for part of +its duration, so 2.18x is the wall-clock delta on this machine, not a clean A/B. +The clean evidence for the padding effect is the 0.06 s vs ~6.4 s short-batch +measurement and the 2.13x estimate on the corpus. + +## Quality panel (frozen 48-query set) + +Control: index built with fixed 512 padding, queries padded to 512. +Treatment: index built with per-batch padding, queries dynamic. +Both run the same hybrid `quality` path and the same text index; only the vector +table and the query padding differ. + +``` +control n=48 hit@1=0.562 hit@5=0.688 MRR=0.601 p50=274ms +treatment n=48 hit@1=0.583 hit@5=0.688 MRR=0.617 p50=258ms +paired: hit@1 +1/-0, hit@5 0/0 (33/48 both), MRR +2/-0 +``` + +No query became worse; one moved into the top-1 and two improved in rank. The +panel is small, so this is "non-inferior on the frozen panel", not a general +quality claim. Command: `experiments/onnx_dense_2026_11/quality_panel.py +--arm control|treatment` with the matching `MSCODEBASE_DATA_DIR`. + +## Correctness + +Per-batch padding changes the tensor width, so quantised inference is not +bit-identical to fixed 512: + +- same text, direct session: cosine 0.996 (12 tokens) / 0.998 (93 tokens); +- 7 059 identical chunks in two indexes: p50 0.998, p5 0.996, p1 0.925, + min 0.863; +- same index, fixed-512 query vs dynamic query: top-1 file identical 14/15, + overlap@5 = 0.48 (vector-only top-5 is unstable on adjacent chunks). + +No retrieval-quality panel was run. `ONNX_FIXED_PADDING=512` restores the old +behaviour for deployments that must match an existing index exactly. + +## Fix + +- `src/core/embedder/onnx_server.py`: pad to the longest text **in the batch**; + keep truncation at 512; `ONNX_FIXED_PADDING` (default 0) restores fixed width. +- Guard: `tests/test_onnx_dynamic_padding.py` (3 cases: per-batch max, truncation, + legacy toggle). +- Verified: dense `smoke_e2e` PASSED (4 checks, step 4 = vector, provider=onnx, + dim=384, results=3). + +## Follow-ups + +1. ~~Run the search-quality panel on a dynamic-padding index before making it + the only mode~~ — done: hit@5 33/48 in both arms, no losing query. +2. Warm/cold build caching: a full rebuild still recomputes every chunk; the + incremental path already skips unchanged files. +3. Consider length-bucketed batching in the indexer instead of sorting by chars + (token-aware), which would tighten batch maxima further. +4. The 48-query panel is small; re-run `experiments/bench_2026_11` on a larger + corpus before removing `ONNX_FIXED_PADDING` from the compatibility surface. diff --git a/experiments/onnx_dense_2026_11/quality_panel.py b/experiments/onnx_dense_2026_11/quality_panel.py new file mode 100644 index 00000000..6c19d552 --- /dev/null +++ b/experiments/onnx_dense_2026_11/quality_panel.py @@ -0,0 +1,187 @@ +r"""Quality panel for the ONNX dynamic-padding change (frozen 48-query panel). + +Control: dense index built with fixed-512 padding (`g4data_onnx`). +Treatment: dense index built with per-batch padding (`g4data_dyn`). +Same frozen queries, same hybrid `quality` path, same text index; only the +vector table and the query embedding mode differ. + +Usage (from the repository root): + $env:MSCODEBASE_DATA_DIR='...\g4data_onnx' + $env:MSCODEBASE_DENSE_OFF='false' + $env:MSCODEBASE_EMBEDDER_PROVIDER='onnx' + python experiments/onnx_dense_2026_11/quality_panel.py --arm control +""" + +from __future__ import annotations + +import argparse +import json +import os +import statistics +import sys +import time +from pathlib import Path + +if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8") + +ROOT = Path(__file__).resolve().parents[2] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) +FROZEN = ROOT / "experiments" / "e57_auto_intent" / "frozen" / "QUERIES.jsonl" +OUT = Path(__file__).resolve().parent + + +class FakeIndexer: + """Minimal indexer surface Searcher needs for the hybrid path.""" + + def __init__(self, project_path: Path): + self.project_path = project_path + self.db_manager = None + self.table = object() + self._symbol_index = None + + async def search_async(self, query_vector, limit: int = 5, filter_expr: str = ""): + """Vector search against the data root in MSCODEBASE_DATA_DIR.""" + import lancedb + + from src.core.artifact_paths import get_db_path + + db = lancedb.connect(str(get_db_path(self.project_path))) + table = db.open_table("codebase_chunks") + builder = table.search(query_vector, vector_column_name="vector") + df = builder.limit(limit).to_pandas() + out = [] + for _, row in df.iterrows(): + out.append( + { + "text": row["text"], + "text_full": row.get("text_full", row["text"]), + "metadata": { + "file": row["file_path"], + "chunk_index": row["chunk_index"], + "indexed_at": row.get("indexed_at", ""), + "layer": row.get("layer", ""), + "hierarchy_level": row.get("hierarchy_level", ""), + "parent_id": row.get("parent_id", ""), + "start_line": row.get("start_line", 0), + "end_line": row.get("end_line", 0), + "symbol_type": row.get("symbol_type", ""), + "module_name": row.get("module_name", ""), + }, + } + ) + return out + + +def _files(rows): + return [ + str((r.get("metadata") or {}).get("file") or "").replace("\\", "/") + for r in rows + ] + + +def _hit(files, gold: str, k: int) -> bool: + return any(f == gold or f.endswith(gold) for f in files[:k]) + + +def _mrr(files, gold: str) -> float: + for rank, f in enumerate(files, 1): + if f == gold or f.endswith(gold): + return 1.0 / rank + return 0.0 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--arm", required=True, help="control | treatment label") + args = parser.parse_args() + + if os.getenv("MSCODEBASE_DENSE_OFF", "true").lower() == "true": + print("REFUSING: MSCODEBASE_DENSE_OFF=true — no vector table would be used") + return 2 + if os.getenv("MSCODEBASE_EMBEDDER_PROVIDER", "").lower() != "onnx": + print("REFUSING: pin MSCODEBASE_EMBEDDER_PROVIDER=onnx") + return 2 + + queries = [ + json.loads(line) + for line in FROZEN.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + + from src.core.search.engine import Searcher + from src.core.search.scoring import auto_detect_intent + from src.providers.embedder.remote_embedder import RemoteEmbedder + + embedder = RemoteEmbedder() + deadline = time.time() + 120 + while time.time() < deadline and not embedder.is_ready(): + time.sleep(2) + if not embedder.is_ready(): + print(f"REFUSING: embedder not ready (mode={getattr(embedder, 'mode', '?')})") + return 2 + print(f"embedder: {embedder.provider_name} dim={embedder.embedding_dim}") + + searcher = Searcher(FakeIndexer(ROOT), embedder) + searcher._ondemand_rerank = False + searcher._multi_reranker_initialized = True + searcher._multi_reranker = None + searcher._demote_non_code = True + searcher._demote_intent_aware = False + searcher._p3_anchor = False + + rows, base_h1, base_h5, mrr, lat = [], [], [], [], [] + for q in queries: + intent = auto_detect_intent(q["query"]) + t0 = time.perf_counter() + code = searcher.search_with_mode(q["query"], mode="quality", limit=5) + t_code = (time.perf_counter() - t0) * 1000 + code_files = _files(code.get("results", []) if isinstance(code, dict) else []) + + t0 = time.perf_counter() + docs_files = _files(searcher.search_docs(q["query"], limit=5)) + t_docs = (time.perf_counter() - t0) * 1000 + + auto_files = docs_files if intent == "docs" else code_files + gold = q["gold_file"] + base_h1.append(_hit(auto_files, gold, 1)) + base_h5.append(_hit(auto_files, gold, 5)) + mrr.append(_mrr(auto_files, gold)) + lat.append(t_docs if intent == "docs" else t_code) + rows.append( + { + "id": q["id"], + "gold": gold, + "intent": intent, + "auto": auto_files, + "h1": base_h1[-1], + "h5": base_h5[-1], + "mrr": mrr[-1], + } + ) + + n = len(queries) + result = { + "arm": args.arm, + "n": n, + "data_dir": os.getenv("MSCODEBASE_DATA_DIR", ""), + "hit@1": sum(base_h1) / n, + "hit@5": sum(base_h5) / n, + "mrr": sum(mrr) / n, + "p50_ms": statistics.median(lat), + "rows": rows, + } + print( + f"{args.arm:<10} n={n} hit@1={result['hit@1']:.3f} " + f"hit@5={result['hit@5']:.3f} MRR={result['mrr']:.3f} " + f"p50={result['p50_ms']:.0f}ms" + ) + (OUT / f"results_{args.arm}.json").write_text( + json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/experiments/onnx_dense_2026_11/results_control.json b/experiments/onnx_dense_2026_11/results_control.json new file mode 100644 index 00000000..16ebb4f5 --- /dev/null +++ b/experiments/onnx_dense_2026_11/results_control.json @@ -0,0 +1,724 @@ +{ + "arm": "control", + "n": 48, + "data_dir": "C:\\Users\\misha\\AppData\\Local\\Temp\\opencode\\g4data_onnx", + "hit@1": 0.5625, + "hit@5": 0.6875, + "mrr": 0.6010416666666667, + "p50_ms": 274.32595000027504, + "rows": [ + { + "id": "E30-E28-01", + "gold": "src/core/rate_limiter.py", + "intent": "auto", + "auto": [ + "src/core/indexing/database_lock.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/database_lock.py", + "src/core/indexing/db_manager.py", + "ISSUE.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-02", + "gold": "src/core/redact.py", + "intent": "auto", + "auto": [ + "src/core/redact.py", + "src/core/redact.py", + "src/core/redact.py", + "src/core/redact.py", + "src/core/redact.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-03", + "gold": "src/core/gitignore_parser.py", + "intent": "auto", + "auto": [ + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-04", + "gold": "src/core/embedder_lease.py", + "intent": "auto", + "auto": [ + "src/core/embedder_lease.py", + "src/core/embedder_lease.py", + "src/core/intelligence/health.py", + "src/core/embedder_lease.py", + "EXPERIMENTS_LOG.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-05", + "gold": "src/core/reindex_ledger.py", + "intent": "auto", + "auto": [ + "src/core/reindex_ledger.py", + "src/core/reindex_ledger.py", + "src/core/reindex_ledger.py", + "src/core/reindex_ledger.py", + "src/core/indexing/index_guard.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-06", + "gold": "src/core/quiet_break_gate.py", + "intent": "auto", + "auto": [ + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-07", + "gold": "src/providers/reranker/reranker_scoring.py", + "intent": "auto", + "auto": [ + "src/providers/reranker/reranker_scoring.py", + "src/providers/reranker/reranker_scoring.py", + "src/providers/reranker/reranker_scoring.py", + "src/providers/reranker/reranker_scoring.py", + "EXPERIMENTS_LOG.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-08", + "gold": "src/core/error_envelope.py", + "intent": "auto", + "auto": [ + "src/core/error_handler.py", + "src/core/error_handler.py", + "src/mcp/server_tools.py", + "src/core/error_handler.py", + "src/core/error_handler.py" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-09", + "gold": "docs/en/GRACEFUL_DEGRADATION.md", + "intent": "docs", + "auto": [ + "docs/ru/GRACEFUL_DEGRADATION.md", + "docs/archive/i18n/ru/ARCHITECTURE_DEEP.md", + "docs/ru/CONFIGURATION.md", + "docs/ru/FAQ.md", + "docs/ru/ARCHITECTURE.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-10", + "gold": "docs/en/SEARCH_PIPELINE.md", + "intent": "docs", + "auto": [ + "docs/ru/TELEMETRY.md", + "docs/ru/README.md", + "docs/ru/SEARCH_PIPELINE.md", + "docs/archive/KNOWN_ISSUES_2026_07.md", + "docs/archive/DEV_DIARY_2026_07.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-11", + "gold": "docs/TRUST_BOUNDARY.md", + "intent": "docs", + "auto": [ + "docs/blog/README.md", + "docs/ru/README.md", + "docs/ru/ZED_WINDOWS_QUIRKS.md", + "docs/archive/research/2026-07-12-e5-base-migration.md", + "docs/TRUST_BOUNDARY.md" + ], + "h1": false, + "h5": true, + "mrr": 0.2 + }, + { + "id": "E30-E28-12", + "gold": "docs/en/TELEMETRY.md", + "intent": "auto", + "auto": [ + "scripts/check_lsp_health.py", + "scripts/summarize_1L_categories.py", + "src/core/artifact_paths.py", + "src/core/artifact_paths.py", + "mscodebase_intelligence.egg-info/SOURCES.txt" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-13", + "gold": "docs/adr/0003-verify-on-read.md", + "intent": "auto", + "auto": [ + "src/core/intelligence/layer.py", + "src/core/doc_llm_verifier.py", + "src/core/intelligence/verify_on_read.py", + "docs/archive/AGENTS.md_before_48_and_audit_20260722_215808.md", + "docs/adr/0003-verify-on-read.md" + ], + "h1": false, + "h5": true, + "mrr": 0.2 + }, + { + "id": "E30-E28-14", + "gold": "docs/en/HANDFOFF.md", + "intent": "docs", + "auto": [ + "docs/archive/i18n/ru/HANDFOFF.md", + "docs/ru/LM_STUDIO_SETUP.md", + "docs/ru/README.md", + "docs/ru/investigations/ACTIVE_WORKSPACE_RESOLUTION.md", + "docs/archive/research/2026-07-12-e5-base-migration.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-15", + "gold": "docs/en/ZED_WINDOWS_QUIRKS.md", + "intent": "docs", + "auto": [ + "docs/ru/investigations/ACTIVE_WORKSPACE_RESOLUTION.md", + "docs/archive/KNOWN_ISSUES_2026_07.md", + "docs/archive/i18n/ru/HANDFOFF.md", + "docs/ru/CLIENTS.md", + "docs/ru/README.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-16", + "gold": "docs/en/SYSTEM_REQUIREMENTS.md", + "intent": "docs", + "auto": [ + "docs/archive/ARCHITECTURE_AUDIT_MCP_vs_IDE.md", + "docs/archive/research/2026-07-09-provider-benchmark.md", + "docs/archive/research/2026-07-12-e5-base-migration.md", + "docs/archive/BENCHMARK.md", + "docs/ru/INSTALL_MODELS.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-17", + "gold": "src/core/indexing/db_writer.py", + "intent": "auto", + "auto": [ + "ISSUE.md", + "AGENT_DIARY.md", + "ISSUE.md", + "AGENT_DIARY.md", + "docs/archive/KNOWN_ISSUES_2026_09.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-18", + "gold": "src/core/artifact_gc.py", + "intent": "auto", + "auto": [ + "src/core/artifact_gc.py", + "src/core/artifact_gc.py", + "src/core/artifact_gc.py", + "src/core/artifact_gc.py", + "docs/en/TELEMETRY.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-19", + "gold": "src/providers/reranker/llama_install.py", + "intent": "auto", + "auto": [ + "src/providers/reranker/llama_install.py", + "src/providers/reranker/llama_install.py", + "src/providers/embedder/remote_embedder.py", + "src/core/embedder/onnx_client.py", + "EXPERIMENTS_LOG.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-20", + "gold": "src/providers/reranker/llama_runner.py", + "intent": "auto", + "auto": [ + "src/core/indexing/database_lock.py", + "src/providers/reranker/llama_runner.py", + "src/providers/reranker/llama_runner.py", + "src/core/indexing/startup_diagnostics.py", + "src/core/indexing/database_lock.py" + ], + "h1": false, + "h5": true, + "mrr": 0.5 + }, + { + "id": "E30-E28-21", + "gold": "src/providers/embedder/remote_embedder.py", + "intent": "auto", + "auto": [ + "src/providers/embedder/remote_embedder.py", + "src/core/embedder_lease.py", + "AGENT_DIARY.md", + "AGENT_DIARY.md", + "docs/archive/DEV_DIARY_2026_07.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-22", + "gold": "src/core/graph.py", + "intent": "auto", + "auto": [ + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-23", + "gold": "src/core/search/cypher_sql.py", + "intent": "auto", + "auto": [ + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-24", + "gold": "src/core/bootstrap_tests.py", + "intent": "auto", + "auto": [ + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-25", + "gold": "src/core/indexing/freshness.py", + "intent": "auto", + "auto": [ + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "scripts/check_zero_vectors.py", + "AGENT_DIARY.md", + "scripts/frozen_overlap_check.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-26", + "gold": "src/core/indexing/parser.py", + "intent": "auto", + "auto": [ + "src/core/indexing/parser.py", + "src/core/indexing/parser.py", + "src/core/indexing/parser.py", + "src/core/indexing/parser.py", + "src/providers/embedder/canary_set.json" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-27", + "gold": "src/core/search/engine.py", + "intent": "auto", + "auto": [ + "src/core/search/scoring.py", + "src/core/search/scoring.py", + "src/core/search/scoring.py", + "scripts/benchmark_search_stages.py", + "src/core/search/scoring.py" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-28", + "gold": "src/core/indexing/db_manager.py", + "intent": "auto", + "auto": [ + "src/core/indexing/indexer_table.py", + "src/core/indexing/indexer_table.py", + "src/core/indexing/index_guard.py", + "src/core/indexing/indexer_table.py", + "src/core/indexing/index_guard.py" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-29", + "gold": "src/core/intelligence/verify_on_read.py", + "intent": "docs", + "auto": [ + "docs/adr/0005-pkg-anchors.md", + "docs/adr/0003-verify-on-read.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-30", + "gold": "src/core/indexing/project_indexer_registry.py", + "intent": "auto", + "auto": [ + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-31", + "gold": "src/mcp/tools/indexing_tools.py", + "intent": "auto", + "auto": [ + "src/sync/server.py", + "src/core/rate_limiter.py", + "src/core/rate_limiter.py", + "src/core/rate_limiter.py", + "src/mcp/tools/indexing_tools.py" + ], + "h1": false, + "h5": true, + "mrr": 0.2 + }, + { + "id": "E30-E28-32", + "gold": "src/providers/reranker/search_result_reranker.py", + "intent": "docs", + "auto": [ + "docs/archive/research/2026-07-12-e5-base-migration.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-33", + "gold": "src/core/search/bm25.py", + "intent": "auto", + "auto": [ + "src/core/search/bm25.py", + "src/core/commit_memory.py", + "src/core/search/bm25.py", + "src/core/search/bm25.py", + "src/core/search/bm25.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D01", + "gold": "src/__init__.py", + "intent": "docs", + "auto": [ + "docs/ru/README.md", + "docs/archive/research/2026-07-11-zed-deep-dive.md", + "docs/adr/0003-verify-on-read.md", + "docs/archive/ARCHITECTURE_AUDIT_MCP_vs_IDE.md", + "docs/ru/GRACEFUL_DEGRADATION.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-D02", + "gold": "src/core/bootstrap_pipeline.py", + "intent": "auto", + "auto": [ + "src/core/bootstrap_pipeline.py", + "src/core/bootstrap_pipeline.py", + "src/core/bootstrap_pipeline.py", + "scripts/monitor.py", + "src/core/bootstrap_pipeline.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D03", + "gold": "src/core/doc_sync_engine.py", + "intent": "auto", + "auto": [ + "src/core/doc_sync_engine.py", + "scripts/negative_controls_runner.py", + "src/core/indexing/symbol_index.py", + "src/core/search/engine.py", + "src/core/indexing/symbol_index.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D04", + "gold": "src/core/gitignore_parser.py", + "intent": "auto", + "auto": [ + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D05", + "gold": "src/core/indexing/freshness.py", + "intent": "auto", + "auto": [ + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D06", + "gold": "src/core/indexing/symbol_index.py", + "intent": "auto", + "auto": [ + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D07", + "gold": "src/core/intelligence/tools_reg.py", + "intent": "auto", + "auto": [ + "src/core/intelligence/tools_reg.py", + "src/core/intelligence/tools_reg.py", + "src/core/intelligence/tools_reg.py", + "src/mcp/server_tools.py", + "src/core/intelligence/tools_reg.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D08", + "gold": "src/core/lsp_project_bridge.py", + "intent": "auto", + "auto": [ + "src/core/lsp_project_bridge.py", + "src/core/system_artifacts.py", + "src/core/lsp_project_bridge.py", + "src/core/system_artifacts.py", + "scripts/check_lsp_health.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D09", + "gold": "src/core/relpath.py", + "intent": "code", + "auto": [ + "src/core/relpath.py", + "src/core/relpath.py", + "src/core/indexing/symbol_index.py", + "src/core/search/evidence.py", + "src/core/search/graph_resolver.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D10", + "gold": "src/core/search/cypher_parser.py", + "intent": "auto", + "auto": [ + "src/core/search/cypher_sql.py", + "src/core/search/cypher_executor.py", + "src/core/search/cypher_engine.py", + "src/core/search/cypher_parser.py", + "src/core/search/cypher_lexer.py" + ], + "h1": false, + "h5": true, + "mrr": 0.25 + }, + { + "id": "E30-D11", + "gold": "src/core/search/trace.py", + "intent": "auto", + "auto": [ + "src/core/search/trace.py", + "src/core/notification_broker.py", + "src/remote_main.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D12", + "gold": "src/mcp/__init__.py", + "intent": "docs", + "auto": [ + "docs/archive/ARCHITECTURE_2026-07-20.md", + "docs/ru/investigations/ACTIVE_WORKSPACE_RESOLUTION.md", + "docs/archive/i18n/ru/ARCHITECTURE_DEEP.md", + "docs/archive/i18n/ru/HANDFOFF.md", + "docs/ru/CLIENTS.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-D13", + "gold": "src/mcp/tools/duplication_tool.py", + "intent": "auto", + "auto": [ + "src/mcp/tools/duplication_tool.py", + "src/core/duplication.py", + "src/mcp/tools/duplication_tool.py", + "src/mcp/tools/duplication_tool.py", + "src/core/duplication.py", + "src/core/duplication.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D14", + "gold": "src/plugins/loader.py", + "intent": "auto", + "auto": [ + "src/plugins/loader.py", + "src/plugins/loader.py", + "examples/plugins/verify_claim/plugin.py", + "src/plugins/manifest.py", + "src/plugins/loader.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D15", + "gold": "src/remote_main.py", + "intent": "auto", + "auto": [ + "src/sync/server.py", + "src/remote_main.py", + "src/sync/server.py", + "src/sync/server.py", + "src/sync/server.py" + ], + "h1": false, + "h5": true, + "mrr": 0.5 + } + ] +} \ No newline at end of file diff --git a/experiments/onnx_dense_2026_11/results_treatment.json b/experiments/onnx_dense_2026_11/results_treatment.json new file mode 100644 index 00000000..dbf7200f --- /dev/null +++ b/experiments/onnx_dense_2026_11/results_treatment.json @@ -0,0 +1,725 @@ +{ + "arm": "treatment", + "n": 48, + "data_dir": "C:\\Users\\misha\\AppData\\Local\\Temp\\opencode\\g4data_dyn", + "hit@1": 0.5833333333333334, + "hit@5": 0.6875, + "mrr": 0.6166666666666667, + "p50_ms": 258.11570000041684, + "rows": [ + { + "id": "E30-E28-01", + "gold": "src/core/rate_limiter.py", + "intent": "auto", + "auto": [ + "src/core/indexing/database_lock.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/database_lock.py", + "src/core/indexing/db_manager.py", + "ISSUE.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-02", + "gold": "src/core/redact.py", + "intent": "auto", + "auto": [ + "src/core/redact.py", + "src/core/redact.py", + "src/core/redact.py", + "src/core/redact.py", + "src/core/redact.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-03", + "gold": "src/core/gitignore_parser.py", + "intent": "auto", + "auto": [ + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-04", + "gold": "src/core/embedder_lease.py", + "intent": "auto", + "auto": [ + "src/core/embedder_lease.py", + "src/core/embedder_lease.py", + "src/core/intelligence/health.py", + "src/core/embedder_lease.py", + "EXPERIMENTS_LOG.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-05", + "gold": "src/core/reindex_ledger.py", + "intent": "auto", + "auto": [ + "src/core/reindex_ledger.py", + "src/core/reindex_ledger.py", + "src/core/reindex_ledger.py", + "src/core/reindex_ledger.py", + "src/core/indexing/index_guard.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-06", + "gold": "src/core/quiet_break_gate.py", + "intent": "auto", + "auto": [ + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py", + "src/core/quiet_break_gate.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-07", + "gold": "src/providers/reranker/reranker_scoring.py", + "intent": "auto", + "auto": [ + "src/providers/reranker/reranker_scoring.py", + "src/providers/reranker/reranker_scoring.py", + "src/providers/reranker/reranker_scoring.py", + "src/providers/reranker/reranker_scoring.py", + "EXPERIMENTS_LOG.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-08", + "gold": "src/core/error_envelope.py", + "intent": "auto", + "auto": [ + "src/core/error_handler.py", + "src/core/error_handler.py", + "src/mcp/server_tools.py", + "src/core/error_handler.py", + "src/core/error_handler.py" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-09", + "gold": "docs/en/GRACEFUL_DEGRADATION.md", + "intent": "docs", + "auto": [ + "docs/ru/GRACEFUL_DEGRADATION.md", + "docs/archive/i18n/ru/ARCHITECTURE_DEEP.md", + "docs/ru/CONFIGURATION.md", + "docs/ru/FAQ.md", + "docs/ru/ARCHITECTURE.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-10", + "gold": "docs/en/SEARCH_PIPELINE.md", + "intent": "docs", + "auto": [ + "docs/ru/TELEMETRY.md", + "docs/ru/README.md", + "docs/ru/SEARCH_PIPELINE.md", + "docs/archive/KNOWN_ISSUES_2026_07.md", + "docs/archive/DEV_DIARY_2026_07.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-11", + "gold": "docs/TRUST_BOUNDARY.md", + "intent": "docs", + "auto": [ + "docs/blog/README.md", + "docs/ru/README.md", + "docs/ru/ZED_WINDOWS_QUIRKS.md", + "docs/archive/research/2026-07-12-e5-base-migration.md", + "docs/TRUST_BOUNDARY.md" + ], + "h1": false, + "h5": true, + "mrr": 0.2 + }, + { + "id": "E30-E28-12", + "gold": "docs/en/TELEMETRY.md", + "intent": "auto", + "auto": [ + "scripts/check_lsp_health.py", + "scripts/summarize_1L_categories.py", + "src/core/artifact_paths.py", + "src/core/artifact_paths.py", + "mscodebase_intelligence.egg-info/SOURCES.txt" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-13", + "gold": "docs/adr/0003-verify-on-read.md", + "intent": "auto", + "auto": [ + "src/core/intelligence/layer.py", + "src/core/doc_llm_verifier.py", + "src/core/intelligence/verify_on_read.py", + "docs/archive/AGENTS.md_before_48_and_audit_20260722_215808.md", + "docs/adr/0003-verify-on-read.md" + ], + "h1": false, + "h5": true, + "mrr": 0.2 + }, + { + "id": "E30-E28-14", + "gold": "docs/en/HANDFOFF.md", + "intent": "docs", + "auto": [ + "docs/archive/i18n/ru/HANDFOFF.md", + "docs/ru/LM_STUDIO_SETUP.md", + "docs/ru/README.md", + "docs/ru/investigations/ACTIVE_WORKSPACE_RESOLUTION.md", + "docs/archive/research/2026-07-12-e5-base-migration.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-15", + "gold": "docs/en/ZED_WINDOWS_QUIRKS.md", + "intent": "docs", + "auto": [ + "docs/ru/investigations/ACTIVE_WORKSPACE_RESOLUTION.md", + "docs/archive/KNOWN_ISSUES_2026_07.md", + "docs/archive/i18n/ru/HANDFOFF.md", + "docs/ru/CLIENTS.md", + "docs/ru/README.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-16", + "gold": "docs/en/SYSTEM_REQUIREMENTS.md", + "intent": "docs", + "auto": [ + "docs/archive/ARCHITECTURE_AUDIT_MCP_vs_IDE.md", + "docs/archive/research/2026-07-09-provider-benchmark.md", + "docs/archive/research/2026-07-12-e5-base-migration.md", + "docs/archive/BENCHMARK.md", + "docs/ru/INSTALL_MODELS.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-17", + "gold": "src/core/indexing/db_writer.py", + "intent": "auto", + "auto": [ + "ISSUE.md", + "AGENT_DIARY.md", + "ISSUE.md", + "AGENT_DIARY.md", + "docs/archive/KNOWN_ISSUES_2026_09.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-18", + "gold": "src/core/artifact_gc.py", + "intent": "auto", + "auto": [ + "src/core/artifact_gc.py", + "src/core/artifact_gc.py", + "src/core/artifact_gc.py", + "src/core/artifact_gc.py", + "docs/en/TELEMETRY.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-19", + "gold": "src/providers/reranker/llama_install.py", + "intent": "auto", + "auto": [ + "src/providers/reranker/llama_install.py", + "src/providers/reranker/llama_install.py", + "src/providers/embedder/remote_embedder.py", + "src/core/embedder/onnx_client.py", + "EXPERIMENTS_LOG.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-20", + "gold": "src/providers/reranker/llama_runner.py", + "intent": "auto", + "auto": [ + "src/core/indexing/database_lock.py", + "src/providers/reranker/llama_runner.py", + "src/providers/reranker/llama_runner.py", + "src/core/indexing/startup_diagnostics.py", + "src/core/indexing/database_lock.py" + ], + "h1": false, + "h5": true, + "mrr": 0.5 + }, + { + "id": "E30-E28-21", + "gold": "src/providers/embedder/remote_embedder.py", + "intent": "auto", + "auto": [ + "src/providers/embedder/remote_embedder.py", + "src/core/embedder_lease.py", + "AGENT_DIARY.md", + "AGENT_DIARY.md", + "docs/archive/DEV_DIARY_2026_07.md" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-22", + "gold": "src/core/graph.py", + "intent": "auto", + "auto": [ + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py", + "src/core/graph.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-23", + "gold": "src/core/search/cypher_sql.py", + "intent": "auto", + "auto": [ + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py", + "src/core/search/cypher_sql.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-24", + "gold": "src/core/bootstrap_tests.py", + "intent": "auto", + "auto": [ + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py", + "src/core/bootstrap_tests.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-25", + "gold": "src/core/indexing/freshness.py", + "intent": "auto", + "auto": [ + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "scripts/check_zero_vectors.py", + "AGENT_DIARY.md", + "scripts/frozen_overlap_check.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-26", + "gold": "src/core/indexing/parser.py", + "intent": "auto", + "auto": [ + "src/core/indexing/parser.py", + "src/core/indexing/parser.py", + "src/core/indexing/parser.py", + "src/core/indexing/parser.py", + "src/providers/embedder/canary_set.json" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-27", + "gold": "src/core/search/engine.py", + "intent": "auto", + "auto": [ + "src/core/search/scoring.py", + "src/core/search/scoring.py", + "src/core/search/scoring.py", + "scripts/benchmark_search_stages.py", + "src/core/search/scoring.py" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-28", + "gold": "src/core/indexing/db_manager.py", + "intent": "auto", + "auto": [ + "src/core/indexing/indexer_table.py", + "src/core/indexing/indexer_table.py", + "src/core/indexing/index_guard.py", + "src/core/indexing/indexer_table.py", + "src/core/indexing/index_guard.py" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-29", + "gold": "src/core/intelligence/verify_on_read.py", + "intent": "docs", + "auto": [ + "docs/adr/0005-pkg-anchors.md", + "docs/adr/0003-verify-on-read.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-30", + "gold": "src/core/indexing/project_indexer_registry.py", + "intent": "auto", + "auto": [ + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py", + "src/core/indexing/project_indexer_registry.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-E28-31", + "gold": "src/mcp/tools/indexing_tools.py", + "intent": "auto", + "auto": [ + "src/sync/server.py", + "src/core/rate_limiter.py", + "src/core/rate_limiter.py", + "src/core/rate_limiter.py", + "src/mcp/tools/indexing_tools.py" + ], + "h1": false, + "h5": true, + "mrr": 0.2 + }, + { + "id": "E30-E28-32", + "gold": "src/providers/reranker/search_result_reranker.py", + "intent": "docs", + "auto": [ + "docs/archive/research/2026-07-12-e5-base-migration.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-E28-33", + "gold": "src/core/search/bm25.py", + "intent": "auto", + "auto": [ + "src/core/search/bm25.py", + "src/core/commit_memory.py", + "src/core/search/bm25.py", + "src/core/search/bm25.py", + "src/core/search/bm25.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D01", + "gold": "src/__init__.py", + "intent": "docs", + "auto": [ + "docs/ru/README.md", + "docs/archive/research/2026-07-11-zed-deep-dive.md", + "docs/adr/0003-verify-on-read.md", + "docs/archive/ARCHITECTURE_AUDIT_MCP_vs_IDE.md", + "docs/ru/GRACEFUL_DEGRADATION.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-D02", + "gold": "src/core/bootstrap_pipeline.py", + "intent": "auto", + "auto": [ + "src/core/bootstrap_pipeline.py", + "src/core/bootstrap_pipeline.py", + "src/core/bootstrap_pipeline.py", + "scripts/monitor.py", + "src/core/bootstrap_pipeline.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D03", + "gold": "src/core/doc_sync_engine.py", + "intent": "auto", + "auto": [ + "src/core/doc_sync_engine.py", + "scripts/negative_controls_runner.py", + "src/core/search/engine.py", + "src/core/indexing/symbol_index.py", + "src/core/search/engine.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D04", + "gold": "src/core/gitignore_parser.py", + "intent": "auto", + "auto": [ + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py", + "src/core/gitignore_parser.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D05", + "gold": "src/core/indexing/freshness.py", + "intent": "auto", + "auto": [ + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py", + "src/core/indexing/freshness.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D06", + "gold": "src/core/indexing/symbol_index.py", + "intent": "auto", + "auto": [ + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py", + "src/core/indexing/symbol_index.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D07", + "gold": "src/core/intelligence/tools_reg.py", + "intent": "auto", + "auto": [ + "src/core/intelligence/tools_reg.py", + "src/core/intelligence/tools_reg.py", + "src/core/intelligence/tools_reg.py", + "src/core/intelligence/tools_reg.py", + "src/core/intelligence/tools_reg.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D08", + "gold": "src/core/lsp_project_bridge.py", + "intent": "auto", + "auto": [ + "src/core/lsp_project_bridge.py", + "src/core/system_artifacts.py", + "src/core/lsp_project_bridge.py", + "src/core/system_artifacts.py", + "scripts/check_lsp_health.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D09", + "gold": "src/core/relpath.py", + "intent": "code", + "auto": [ + "src/core/relpath.py", + "src/core/relpath.py", + "src/core/indexing/symbol_index.py", + "src/core/search/evidence.py", + "src/core/search/graph_resolver.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D10", + "gold": "src/core/search/cypher_parser.py", + "intent": "auto", + "auto": [ + "src/core/search/cypher_sql.py", + "src/core/search/cypher_parser.py", + "src/core/search/cypher_engine.py", + "src/core/search/cypher_executor.py", + "src/core/search/cypher_lexer.py" + ], + "h1": false, + "h5": true, + "mrr": 0.5 + }, + { + "id": "E30-D11", + "gold": "src/core/search/trace.py", + "intent": "auto", + "auto": [ + "src/core/search/trace.py", + "src/core/notification_broker.py", + "src/remote_main.py", + "src/mcp/tools/search_tools.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D12", + "gold": "src/mcp/__init__.py", + "intent": "docs", + "auto": [ + "docs/archive/ARCHITECTURE_2026-07-20.md", + "docs/ru/investigations/ACTIVE_WORKSPACE_RESOLUTION.md", + "docs/archive/i18n/ru/ARCHITECTURE_DEEP.md", + "docs/archive/i18n/ru/HANDFOFF.md", + "docs/ru/CLIENTS.md" + ], + "h1": false, + "h5": false, + "mrr": 0.0 + }, + { + "id": "E30-D13", + "gold": "src/mcp/tools/duplication_tool.py", + "intent": "auto", + "auto": [ + "src/mcp/tools/duplication_tool.py", + "src/core/duplication.py", + "src/mcp/tools/duplication_tool.py", + "src/mcp/tools/duplication_tool.py", + "src/core/duplication.py", + "src/core/duplication.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D14", + "gold": "src/plugins/loader.py", + "intent": "auto", + "auto": [ + "src/plugins/loader.py", + "src/plugins/loader.py", + "examples/plugins/verify_claim/plugin.py", + "src/plugins/loader.py", + "src/plugins/manifest.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + }, + { + "id": "E30-D15", + "gold": "src/remote_main.py", + "intent": "auto", + "auto": [ + "src/remote_main.py", + "src/sync/server.py", + "src/sync/server.py", + "src/sync/server.py", + "src/sync/server.py" + ], + "h1": true, + "h5": true, + "mrr": 1.0 + } + ] +} \ No newline at end of file diff --git a/src/core/embedder/onnx_server.py b/src/core/embedder/onnx_server.py index 6fceecd5..62206c92 100644 --- a/src/core/embedder/onnx_server.py +++ b/src/core/embedder/onnx_server.py @@ -37,6 +37,20 @@ _shutdown_event = threading.Event() +def _fixed_pad_length() -> int: + """Optional legacy mode: pad every text to this length (0 = per batch). + + Some deployments want vectors bit-identical to builds made before the + per-batch padding change (2026-11-12). `ONNX_FIXED_PADDING=512` restores + that; default is dynamic padding, which keeps a batch's cost proportional + to its longest text. + """ + try: + return max(0, int(os.getenv("ONNX_FIXED_PADDING", "0") or 0)) + except ValueError: + return 0 + + class OnnxHandler(BaseHTTPRequestHandler): """HTTP handler для эмбеддинг API.""" @@ -103,16 +117,15 @@ def _embed_single(self, text: str) -> List[float]: # Токенизация через tokenizers (как в RemoteEmbedder) enc = tokenizer.encode(text, add_special_tokens=True) - # Padding/truncation до max_length + # Truncation только; padding для одного текста не нужен (G4 perf). ids = enc.ids[:max_length] attention_mask = enc.attention_mask[:max_length] - if len(ids) < max_length: - pad_len = max_length - len(ids) - ids = ids + [1] * pad_len # pad_id = 1 - attention_mask = attention_mask + [0] * pad_len # Подготовка входов для ONNX - seq_len = len(ids) + seq_len = max(len(ids), _fixed_pad_length()) + if seq_len > len(ids): + ids = ids + [1] * (seq_len - len(ids)) + attention_mask = attention_mask + [0] * (seq_len - len(attention_mask)) onnx_inputs = {} for name in input_names: if name == "input_ids": @@ -159,17 +172,25 @@ def _embed_batch(self, texts: List[str]) -> List[List[float]]: for enc in encodings: ids = enc.ids[:max_length] mask = enc.attention_mask[:max_length] - if len(ids) < max_length: - pad_len = max_length - len(ids) - ids = ids + [1] * pad_len # pad_id = 1 - mask = mask + [0] * pad_len batch_ids.append(ids) batch_masks.append(mask) + # Pad to the LONGEST text in THIS batch, not to 512. The indexer sorts + # chunks by length, so a batch is homogeneous; fixed 512 padding made a + # 31-token chunk cost exactly as much as a 500-token one (measured + # 2026-11-12: every 32-text batch took ~6-8 s regardless of length). + # ONNX_FIXED_PADDING restores the legacy width for old-index parity. + max_len = max((len(ids) for ids in batch_ids), default=0) + if max_len == 0: + return [] + seq_len = max(max_len, _fixed_pad_length()) + pad_id = 1 + batch_ids = [ids + [pad_id] * (seq_len - len(ids)) for ids in batch_ids] + batch_masks = [mask + [0] * (seq_len - len(mask)) for mask in batch_masks] + batch_array = np.array(batch_ids, dtype=np.int64) mask_array = np.array(batch_masks, dtype=np.int64) batch_size = len(batch_ids) - seq_len = batch_array.shape[1] onnx_inputs = {} for name in input_names: if name == "input_ids": @@ -285,9 +306,9 @@ def load_model(model_name: str): if not tokenizer_path.exists(): raise FileNotFoundError(f"Tokenizer not found: {tokenizer_path}") - # Загружаем токенизатор + # Загружаем токенизатор. Padding — per-batch (в _embed_batch), не fixed + # 512: иначе короткий чанк стоит как 512-токенный (замер 2026-11-12). tokenizer = Tokenizer.from_file(str(tokenizer_path)) - tokenizer.enable_padding(pad_token="", pad_id=1, length=512) tokenizer.enable_truncation(max_length=512) # Настройка ONNX Runtime diff --git a/tests/test_onnx_dynamic_padding.py b/tests/test_onnx_dynamic_padding.py new file mode 100644 index 00000000..1ff9e35a --- /dev/null +++ b/tests/test_onnx_dynamic_padding.py @@ -0,0 +1,63 @@ +"""ONNX server pads per batch, not to a fixed 512 (dense-build perf, 2026-11-12). + +Measured before the fix: every 32-text batch took 6-8 s whether texts were 30 +or 2000 chars, because the tokenizer padded all of them to 512 tokens. The +indexer sorts chunks by length, so a per-batch maximum is nearly the exact +length and removes the waste. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import numpy as np + +from src.core.embedder.onnx_server import OnnxHandler + + +class _FakeSession: + def __init__(self): + self.seen_seq_len = None + self.seen_batch = None + + def run(self, _outputs, inputs): + self.seen_batch, self.seen_seq_len = inputs["input_ids"].shape + return [np.zeros((self.seen_batch, self.seen_seq_len, 384), dtype=np.float32)] + + +def _handler(ids_list): + handler = OnnxHandler.__new__(OnnxHandler) + encodings = [ + SimpleNamespace(ids=ids, attention_mask=[1] * len(ids)) for ids in ids_list + ] + handler.server = SimpleNamespace( + session=_FakeSession(), + tokenizer=SimpleNamespace(encode_batch=lambda _texts, add_special_tokens: encodings), + input_names=["input_ids", "attention_mask", "token_type_ids"], + max_length=512, + ) + return handler + + +def test_batch_pads_to_the_longest_text_not_to_512(): + handler = _handler([[1, 2, 3], [4] * 40, [5] * 7]) + handler._embed_batch(["a", "b", "c"]) + assert handler.server.session.seen_seq_len == 40, ( + "batch must be padded to its own maximum length, not 512" + ) + assert handler.server.session.seen_batch == 3 + + +def test_batch_truncates_at_max_length(): + handler = _handler([[1] * 900]) + handler._embed_batch(["long"]) + assert handler.server.session.seen_seq_len == 512 + + +def test_fixed_padding_toggle_restores_legacy_width(monkeypatch): + monkeypatch.setenv("ONNX_FIXED_PADDING", "512") + handler = _handler([[1, 2, 3], [4] * 40]) + handler._embed_batch(["a", "b"]) + assert handler.server.session.seen_seq_len == 512, ( + "ONNX_FIXED_PADDING=512 must restore the old 512-wide inputs" + )