fix(benchmark): 对齐官方 Judge 与 LongMemEval con 回答流程

agent_memory_leaderboard/benchmarks/EVALUATION.md

@@ -7,24 +7,84 @@ OpenAI-compatible answer call and a separate dataset-aware judge call for
 LoCoMo-Refined and LongMemEval-S. Both modes are local diagnostics: neither
 consumes AML Full quota nor produces an official AML score.
 
-The local judge uses versioned, independently maintained rubrics inspired by
+The self-run judge uses the complete, pinned upstream prompts from
 [LoCoMo-Refined](https://github.com/mem-eval-suite/LoCoMo_refined/tree/887091190789e8d6760e70b9edd696539923dc4f)
 and [LongMemEval](https://github.com/xiaowu0162/LongMemEval/tree/9e0b455f4ef0e2ab8f2e582289761153549043fc).
-It does **not** reproduce those repositories' native evaluator prompts, output
-handling or model runtimes. Reports set `native_evaluator_equivalent=false`.
-Use a fixed answer model, judge model and dataset for comparisons; choosing
-Qwen3-14B alone does not make this local judge equivalent to the refined judge.
-Native evaluation and official AML validation remain separate acceptance work.
+LoCoMo uses the unmodified `refined` template, one request for each acceptable
+gold candidate, and the maximum candidate score. LongMemEval uses its original
+category-specific `get_anscheck_prompt`, including abstention. Sources, revisions,
+prompt hashes, request options, aggregation and the scorer version are recorded
+in the evaluation contract. The HTTP client retains bounded retries/timeouts
+rather than copying upstream's runtime: LoCoMo upstream retries evaluation errors
+up to ten times, while this client retries transient transport/status failures
+up to four times and leaves malformed JSON pending for explicit resume. Compare
+coverage separately when failures occur. Vendored material and its attribution/license notices
+are under `src/judges/`.
+
+Use Qwen3-14B for the LoCoMo-Refined judge. LongMemEval's pinned evaluator supports
+`gpt-4o-2024-08-06`, `gpt-4o-mini-2024-07-18`, and
+`meta-llama/Meta-Llama-3.1-70B-Instruct`; select and record the intended model.
+A different configured model produces a warning and
+`judge_model_matches_upstream=false` in each evaluation. Model aliases do not
+verify the weights deployed by a provider.
+
+Official prompts alone do **not** reproduce the complete native benchmark:
+retrieval/chunking, answer generation, context selection, lexical metrics and
+provider runtime remain this client's implementation. Reports therefore retain
+`native_evaluator_equivalent=false`. Use a fixed answer model, judge model and
+dataset for comparisons. Official AML validation is separate acceptance work.
 
 "Local" means a self-run evaluation, including GitHub Actions; it does not
 require a locally deployed Memoria service. The workflow and CLI share this
 evaluator. AML's public documentation describes a platform-controlled Answer/Eval
 flow, but the complete official judge prompts and a reproducible implementation
 were not found in the documentation checked on 2026-10-07. The prompts in
-`src/evaluation.py` are maintained by this project, not copied from AML.
+`src/judges/` come from the public benchmark repositories, not from AML.
 See the [official documentation](https://agentmemories.ai/zh-cn/docs) and
 [competition readiness plan](../docs/competition-plan.md#正式参评前的工作).
 
+## LongMemEval answer reader
+
+LongMemEval now defaults to its upstream `con` reading method: a single answer
+request using the complete step-by-step extraction/reasoning template. This is
+`--cot true` in the upstream shell script, not the separately executed notes
+extraction of `con-separate`. Set `BENCHMARK_LONGMEMEVAL_READING_METHOD=direct`
+for the original direct template. `con-separate` is not implemented.
+
+Both templates are pinned to the same upstream revision as the judge. The reader
+uses one user message, `n=1`, temperature 0, the question date as `Current Date`,
+and dated JSON session blocks retaining both user and assistant content. Default
+answer output is 800 tokens for `con`, 500 for `direct`. Explicit
+`BENCHMARK_OUTPUT_TOKENS` overrides these defaults and is recorded in the contract.
+
+Only records actually returned by Search enter the reader. Exact content hashes
+recover date, role, session and turn order from the label-free ingestion data.
+Partial stored fragments stay partial; matching a fragment never restores the
+rest of its original turn or other messages in the session. Ambiguous/unmapped
+records retain their returned text with date/role marked unknown, and the count
+is reported. Gold answers and evidence labels are never used to fill context.
+`top_k` still counts adapter records, not native full sessions.
+
+The selected records are grouped and ordered by session date and turn position.
+History is truncated as a token prefix, reserving output tokens plus 1000 tokens
+as upstream does. An additional full-request cap includes instructions and chat
+serialization. Diagnostics record fully included original record indices and
+partially included indices; within-budget source recall credits only complete
+records instead of assuming retrieval ranks remain a prefix after date sorting.
+
+Without an explicit `BENCHMARK_INPUT_BUDGET`, known answer models use the pinned
+upstream context-window table (for example, GPT-4o 128000). Unknown models require
+an explicit verified limit; a configured limit above a known model's window is
+rejected. This client uses the configured tiktoken encoding; upstream uses model
+specific Hugging Face tokenizers for non-OpenAI readers, so those reader setups
+still require tokenizer alignment. The resolved budgets, reading method and
+reader profile are included in the run contract and answer-cache signature.
+Old checkpoints cannot silently reuse answers generated by a different template.
+
+The Search cache and successful Answer cache remain reusable when only a pending
+Judge call is resumed under the same contract. This change does not start a new
+benchmark, change the deployed adapter, or alter the stored historical results.
+
 ## Opt-in local content diagnostics
 
 Content diagnostics run **in the self-run benchmark client**, on the machine
@@ -92,7 +152,9 @@ are explicitly not proof that a request was sent. Successful request/response
 snapshots come from the execution path, not later database reconstruction.
 
 No invented judge explanation or hidden model reasoning is requested. When the
-judge provides only `correct`, the page states that no explanation was supplied.
+judge provides only a label, the page states that no explanation was supplied.
+For multiple gold candidates, all candidate inputs and outputs are retained;
+the per-question score is their maximum, rather than the final displayed label.
 Source mapping ambiguity, missing annotated sources, budget exclusion and model
 failure are diagnostic signals, not automatic root-cause findings.
 
@@ -232,13 +294,13 @@ reported as an evaluation skip without calling answer/judge models, not scored a
   higher is better. Case-insensitive word/punctuation tokenization.
 - **BLEU-1**: clipped unigram precision with a brevity penalty for LoCoMo; 0–1,
   higher is better. A short answer missing reference information is penalized.
-- **Local judge accuracy**: mean binary correctness under the local dataset
-  rubric; 0–1, higher is better. LME abstention, knowledge update, preference and
+- **Local judge accuracy**: mean binary correctness under the pinned upstream
+  prompt and configured model; 0–1, higher is better. LME abstention, knowledge update, preference and
   temporal questions have distinct rules. It is not a native/public leaderboard score.
 - **Within-budget evidence recall**: source recall using only the ranked prefix
   actually passed to the answer model; 0–1, higher is better.
 
-The default answer input budget is 117760, counting serialized instructions,
+For LoCoMo, the default answer input budget is 117760, counting serialized instructions,
 roles, question, optional options/date and retrieved evidence with the configured
 `tiktoken` encoding, plus a 256-token safety margin. Evidence is retained as a
 whole-record ranked prefix: the first oversized record ends selection; lower
@@ -249,25 +311,37 @@ adapter. A Qwen model may tokenize differently. Report provider prompt usage
 when available; a reported over-budget request fails evaluation. That check is
 after the provider call and does not guarantee avoidance of billing or over-limit
 requests. Confirm the actual model tokenizer/template before formal comparisons.
-Output length has a separate limit (default 1024); temperature is zero.
-
-Prefix selection uses binary search with memoized full-request token counts,
+LoCoMo answer output length defaults to 1024; temperature is zero. LongMemEval
+uses the dataset-specific reader defaults described above.
+Judge settings follow upstream: LoCoMo has no explicit output-token override and
+requests `enable_thinking=false`; LongMemEval uses `n=1` and `max_tokens=10`.
+If a provider explicitly rejects `enable_thinking` as unsupported, it is removed
+and the fallback is recorded as `thinking_option_unsupported`. Other 400/422
+responses do not trigger this fallback.
+
+For LoCoMo, prefix selection uses binary search with memoized full-request token counts,
 reducing encoding passes from O(N) to O(log N) for N retrieved records. Each
 candidate includes all prompt fields and the selected result is checked as a
 complete input; this avoids assuming that BPE counts are additive per record.
 
 ### Model failures and scoring coverage
 
-Answer and judge calls each allow at most four HTTP attempts. Transport errors,
+Answer and judge calls each allow at most four transient HTTP attempts. LoCoMo's
+explicit unsupported-thinking-option fallback can add one HTTP request. Transport errors,
 429 and 5xx use exponential backoff (0.5, 1, 2 seconds); numeric `Retry-After`
 on 429 is capped at 30 seconds per delay. Other HTTP errors are not retried.
 Safe per-question attempt diagnostics include phase, attempt, HTTP status or
 transport marker, duration and retry flag, without provider bodies or credentials.
 
-The judge accepts a strict JSON object containing only boolean `correct`, either
-bare or inside a complete Markdown code fence. Invalid JSON, invalid response
-envelopes, blank output, nonterminal completion reasons including `length`,
-exhausted transient failures and budget overflow fail the affected question.
+LoCoMo parses upstream's JSON `label: CORRECT/WRONG` format, including fenced
+objects. LongMemEval parses `yes/no`. The previous custom `correct: true/false`
+format and abbreviated rubrics are no longer used. Label decisions also follow
+upstream: LoCoMo treats a parsed object whose label is not `CORRECT` as zero;
+LongMemEval checks whether `yes` appears in the response. These are upstream
+behaviors, not newly tightened validation rules. Invalid JSON and malformed
+response envelopes fail evaluation. Answer output must be nonblank and finish
+normally; Judge text is parsed regardless of finish reason, matching upstream.
+Exhausted transient failures and budget overflow also fail the affected question.
 It remains `evaluation_pending` with `answer_failed` or `judge_failed`; later
 questions continue. Authentication/endpoint errors (401/403/404/405) stop the run
 early because they normally affect every question. Storage and programming
@@ -288,7 +362,10 @@ retrying its judge. This avoids repeated Add/Search and answer calls. Interrupte
 calls whose result was not saved may still repeat. Retry, parser and context
 selection versions are recorded in the evaluation contract; pre-hardening
 Answer/Eval checkpoints do not match this contract and require a fresh run or
-their original runner. Retrieval-only checkpoint compatibility is unchanged.
+their original runner. Changing this scorer version likewise prevents resuming
+an old Answer/Eval run with mixed scoring rules. Retrieval-only checkpoint
+compatibility is unchanged. Rejudging only previously incorrect predictions is a
+subset audit; do not report a new full-run accuracy by mixing old and new judges.
 
 ### Checkpoints and privacy
 

agent_memory_leaderboard/benchmarks/evaluation.env.example

@@ -1,14 +1,20 @@
 # Optional; only used with --mode answer-eval. Never commit actual credentials.
 # Answer and Judge can use different OpenAI-compatible providers.
-# Current development baseline: DashScope Beijing qwen-plus for both roles.
-# Local diagnostics only; these settings do not select AML's official models.
+# Answer generation below is a development baseline, not AML's official model.
+# LoCoMo-Refined's upstream judge is Qwen3-14B. For LongMemEval, select a pinned
+# supported judge model and compatible provider; see EVALUATION.md.
 BENCHMARK_ANSWER_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
 BENCHMARK_ANSWER_MODEL=qwen-plus
 BENCHMARK_ANSWER_API_KEY=
 BENCHMARK_JUDGE_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
-BENCHMARK_JUDGE_MODEL=qwen-plus
+BENCHMARK_JUDGE_MODEL=qwen3-14b
 BENCHMARK_JUDGE_API_KEY=
 # Local budget accounting; this encoding is not asserted to match every model.
 BENCHMARK_TOKEN_ENCODING=o200k_base
-BENCHMARK_INPUT_BUDGET=117760
-BENCHMARK_OUTPUT_TOKENS=1024
+# Leave budgets unset to use dataset/model defaults. Explicit values override them.
+# LoCoMo: input 117760, output 1024.
+# LongMemEval: input from the pinned model window table, con output 800/direct 500.
+# Unknown answer models require an explicit verified input budget.
+# BENCHMARK_INPUT_BUDGET=128000
+# BENCHMARK_OUTPUT_TOKENS=800
+BENCHMARK_LONGMEMEVAL_READING_METHOD=con

agent_memory_leaderboard/scripts/content_diagnostics_demo.py

@@ -90,7 +90,7 @@ def respond(request):
             result = (
                 "invalid synthetic verdict"
                 if scenario == "judge-failure"
-                else json.dumps({"correct": scenario == "complete"})
+                else json.dumps({"label": "CORRECT" if scenario == "complete" else "WRONG"})
             )
         return httpx.Response(
             200, json={"choices": [{"message": {"content": result}, "finish_reason": "stop"}]}

agent_memory_leaderboard/src/benchmark.py

@@ -127,7 +127,7 @@ def run(args):
         "runner_version": 2,
         "ingestion_version": "label-free-exact-hash-v1",
         "mode": mode,
-        "evaluation": evaluation_config.contract() if evaluation_config else None,
+        "evaluation": evaluator.config.contract() if evaluator else None,
         "kind": args.kind,
     }
     if diagnostic_policy is not None:
@@ -356,7 +356,14 @@ def call(http, path, body, identity):
                         if content is not None:
                             content.question(qid, sample, question, metrics=record)
                         try:
-                            evaluation = evaluator.evaluate(question, contents, cache_path)
+                            evaluation = evaluator.evaluate(
+                                question,
+                                contents,
+                                cache_path,
+                                reader_records=manifest.reader_records(contents)
+                                if args.kind == "longmemeval"
+                                else None,
+                            )
                         except ModelError as exc:
                             evaluation = exc.partial_result
                             record.update(
@@ -374,7 +381,12 @@ def call(http, path, body, identity):
                         if "context_records" in evaluation:
                             evaluation["budget_evidence_recall"] = recall(
                                 question.get("evidence", []),
-                                ranked_ids[: evaluation["context_records"]],
+                                [
+                                    ranked_ids[index]
+                                    for index in evaluation["context_record_indices"]
+                                ]
+                                if "context_record_indices" in evaluation
+                                else ranked_ids[: evaluation["context_records"]],
                             )
                         record["evaluation"] = evaluation
                         record["model_diagnostics"] = list(evaluator.model_attempts)

agent_memory_leaderboard/src/content_diagnostics.py

@@ -30,6 +30,9 @@
     "context_selection_version",
     "model_retry_version",
     "verdict_parser_version",
+    "longmemeval_reading_method",
+    "longmemeval_reader_profile",
+    "judge_profiles",
 }
 
 
@@ -429,8 +432,12 @@ def model(self, phase, value):
         if phase.endswith("_input"):
             record["model"].pop(phase.replace("_input", "_output"), None)
         if phase == "answer_input":
-            for stale in ("judge_input", "judge_output"):
+            for stale in ("judge_input", "judge_output", "judge_candidates"):
                 record["model"].pop(stale, None)
+        if phase == "judge_input":
+            record["model"].setdefault("judge_candidates", []).append({"input": value})
+        elif phase == "judge_output" and record["model"].get("judge_candidates"):
+            record["model"]["judge_candidates"][-1]["output"] = value
         self.put(self.current, record, "question", record["sample"])
 
     def resume_search(self, contents):
@@ -476,10 +483,19 @@ def render(self, status, summary, error=None):
                     record.get("search_request", "not captured; consult run stage/status"),
                 )
                 included = record.get("model", {}).get("answer_input", {}).get("included_records")
+                answer_input = record.get("model", {}).get("answer_input", {})
+                indices = answer_input.get("context_record_indices")
+                partial = answer_input.get("partial_context_record_indices", [])
                 for rank, item in enumerate(record.get("retrieved", []), 1):
                     selection = (
                         "not evaluated"
                         if included is None
+                        else "partially included by token truncation"
+                        if rank - 1 in partial
+                        else "included"
+                        if indices is not None and rank - 1 in indices
+                        else "excluded by input budget"
+                        if indices is not None
                         else "included"
                         if rank <= included
                         else "excluded by input budget"