Refuse zero-coverage remaps, declare postconditions and response contracts

CHANGELOG.md

@@ -19,6 +19,33 @@ uses Semantic Versioning for public releases.
   values.
 
 ### Added
+- `remap_to_rectilinear` refuses when source coverage is zero, using the same
+  `input_required` payload as the vector preconditions. Every value in a
+  zero-coverage remap is an extrapolation, so a warning beside the numbers was
+  not enough. Partial coverage and a non-conservative method remain warnings.
+- Analysis results carry a `postconditions` block that is explicit about not
+  having checked. `calculate_area` verifies the closed-mesh identity
+  `sum(face_areas) == 4*pi*R^2` and abstains with `not_evaluated` on open or
+  regional meshes rather than reporting a verdict it cannot support.
+- `run_analysis` accepts `verdict_policy` (`full`, `reference_only`, `off`,
+  also readable from `UXARRAY_MCP_VERDICT_POLICY`) so a caller can ask for the
+  reference and tolerance without the server's own verdict. An unrecognized
+  policy is rejected before the computation runs.
+- Two tools under a new `contract/` namespace: `describe_response_contract`
+  declares the fields a named response shape requires, and `validate_response`
+  checks a candidate payload against it. This makes "right answer, wrong
+  envelope" separately detectable instead of scoring as a wrong answer.
+- Physical test fixtures: a global mesh carrying a real Earth `sphere_radius`,
+  a four-level field whose levels are far enough apart that a mis-selection is
+  unmistakable, and a half-masked field where any mean other than 1.0 means
+  NaNs were folded in. Every previous fixture sat on a unit sphere, where
+  radius scaling is invisible.
+- `evals/multi_turn/` measures what one-call benchmarks cannot: whether a run
+  chains the calls a task requires, reuses minted handles instead of inventing
+  or dropping them, and recovers from an injected mid-sequence fault. Two
+  injected faults, a precondition refusal and an interrupted workflow, give the
+  refusal machinery something to be validated against. Two scripted adapters
+  bracket the score range so the harness runs offline.
 - `curl` and `divergence` declare their preconditions as data and refuse
   instead of returning an unphysical number. The refusal is shaped after the
   MCP `2026-07-28` multi-round-trip request flow: `result_type:

artifacts/facility_matrix/native_paths.example.json

@@ -0,0 +1,5 @@
+{
+  "chrysalis": {"grid_path": "/facility/path/to/grid.nc", "label": "Chrysalis native MPAS mesh"},
+  "improv": {"grid_path": "/facility/path/to/grid.nc", "label": "Improv native MPAS mesh"},
+  "ucar": {"grid_path": "/facility/path/to/grid.nc", "label": "Casper native UGRID mesh"}
+}

artifacts/facility_matrix/native_paths.json

@@ -0,0 +1,14 @@
+{
+  "chrysalis": {
+    "grid_path": "/lcrc/group/e3sm/data/inputdata/ocn/mpas-o/oQU120/oQU120.grid.nc",
+    "label": "oQU120 MPAS-O production grid"
+  },
+  "improv": {
+    "grid_path": "/gpfs/fs1/home/jain/uxarray/test/meshfiles/mpas/QU/480/grid.nc",
+    "label": "QU/480 MPAS grid"
+  },
+  "ucar-uxarray-yac": {
+    "grid_path": "/glade/u/home/rajeevj/uxarray/test/meshfiles/mpas/QU/mesh.QU.1920km.151026.nc",
+    "label": "QU/1920km MPAS grid"
+  }
+}

docs/issues-from-escience-study/01-run-analysis-is-too-general.md

@@ -0,0 +1,66 @@
+# `run_analysis` is a single tool with 38 parameters and 32 operations
+
+## Problem
+
+`src/uxarray_mcp/tools/frontdoor.py` exposes essentially the whole library
+through one function. As of `de21d322`:
+
+- **38 parameters** on `run_analysis`
+- **32 operations** named in a **1210-character** docstring
+- `get_capabilities` adds a further 2923-character description
+
+Serialized, the two tools come to about **9,100 characters** of tool
+specification sent on every request. The comparison condition in the study,
+a plain file inspector plus a Python interpreter, was **619 characters**.
+
+That 15x difference is not inherent to MCP. It is a consequence of this
+design choice, and the paper says so explicitly rather than attributing the
+overhead to the protocol.
+
+## Why it matters beyond size
+
+A tool this general cannot say anything specific about a particular call. It
+cannot express that `calculate_zonal_mean` needs face-centered data while
+`calculate_area` needs no data at all, because it has one signature for all 32
+operations. Most of the 38 parameters are irrelevant to any given call
+(`center_lon` and `outer_radius` mean nothing to `calculate_area`), so the
+model must infer applicability from prose.
+
+This also appears to drive issue 02: because the tool cannot describe the
+specific call, it compensates by attaching a catalog of everything else it
+could do.
+
+## Evidence
+
+Measured directly from the archived schemas:
+
+```
+persistent_generic  tools=[inspect_dataset, run_python]        schema =   619 chars
+operation_only_mcp  tools=[get_capabilities, run_analysis]     schema = 9,100 chars
+```
+
+On the two easy control tasks the named-operation interface cost about
+**3.2k and 3.8k more median tokens** than plain Python, for identical
+scientific outcomes (20/20 either way). That is pure overhead on decisions
+that were easy to begin with.
+
+Note the honest counterweight: on the remap task the named operation was worth
+it, taking 10/20 to 20/20 (`p = 4e-4`) and cutting median tokens for three of
+four deployments. The problem is not having named operations. It is having one
+name for everything.
+
+## Suggested direction
+
+Split the front door into a small number of tools whose schemas can be
+specific, for example grouped by what they consume: mesh-only operations
+(`calculate_area`, `inspect_mesh`), single-field operations
+(`calculate_zonal_mean`, `gradient`), two-field operations (`curl`,
+`divergence`, `bias`), and subsetting or export. Each gets only the parameters
+it can use.
+
+If the single entry point has to stay for compatibility, consider generating
+per-operation schemas and advertising only the subset applicable to the dataset
+currently in play.
+
+Worth measuring after any change: serialized schema size, and whether
+first-turn prompt tokens fall from the observed median of 2,000.

docs/issues-from-escience-study/02-capability-catalog-dominates-every-reply.md

@@ -0,0 +1,65 @@
+# A tool catalog is 74% of what the server sends back
+
+## Problem
+
+This is the study's clearest finding about the server, and it was a surprise to
+us. We added status, warnings, advice, and provenance to results expecting it to
+help models make better scientific decisions. It did not help at all on the task
+it was designed for. When we measured what we had actually been sending, the
+reason was obvious.
+
+Summing serialized bytes per top-level key across all 120 runs of the enriched
+condition:
+
+| What we actually sent | Share |
+|---|---|
+| `mcp_server_tools` | 25.9% |
+| `variables` | 24.2% |
+| `_provenance` | 16.2% |
+| `recommended_next_steps` | 13.8% |
+| `uxarray_capabilities` | 8.1% |
+| `recommendations` | 3.5% |
+| `scientific_status` | **1.9%** |
+| `postconditions` | 1.8% |
+| everything else, including the computed numbers | ~4.6% |
+
+Grouped: **74.1% is a catalog of other tools and capabilities**, 16.2% is a
+provenance record, 7.5% is status and warnings, and **2.2% is the actual
+result**.
+
+On the vector-suitability task the field that settles the scientific question,
+`scientific_status`, was **100 bytes inside a 2,172-byte reply — 4.6%**.
+
+## Effect
+
+The enriched results roughly doubled that task's median token use, from about
+14.0k to 28.6k, and changed the outcome not at all: 10/20 with and without,
+Fisher exact `p = 1.0`. This is a real null, not an underpowered one.
+
+The correct reading is not that scientific evidence is useless. It is that a
+decisive signal at 2% of the payload, surrounded by a tool catalog, does not
+change behavior.
+
+## Where it comes from
+
+`src/uxarray_mcp/tools/capabilities.py` builds `mcp_server_tools`,
+`uxarray_capabilities`, `recommended_next_steps`, and `endpoint_profiles`. These
+are useful once, during discovery. They are attached to results that a model has
+already decided to request, and then re-sent on every subsequent turn because
+the conversation carries them forward.
+
+## Suggested direction
+
+Separate discovery from results. A result should describe *that result*.
+
+- Keep `get_capabilities` as the discovery call, and let it be as verbose as it
+  needs to be, since it is called once.
+- Remove `mcp_server_tools`, `uxarray_capabilities`, `recommended_next_steps`,
+  and `endpoint_profiles` from analysis results entirely.
+- Consider putting `_provenance` behind a handle. It is 16% and is rarely what
+  the model needs for its next decision, though it does matter for the
+  scientific record, so this needs thought rather than deletion.
+- Keep `variables` only when the operation is an inspection.
+
+A rough target: the computed numbers plus status should be the majority of a
+result payload, not 4%.

docs/issues-from-escience-study/03-results-cannot-say-whether-anything-was-checked.md

@@ -0,0 +1,77 @@
+# Results cannot say whether anything was checked
+
+## Problem
+
+Nothing in `src/uxarray_mcp/` currently distinguishes "this ran successfully"
+from "this answer was compared against something known." Grep for
+`postconditions`, `not_evaluated`, or `scientific_status` in `src/` and there
+are no hits. Those fields existed only in the experiment patch
+(`uxarray_mcp_frontdoor_v3.patch`) and were never merged.
+
+This matters because it is the one change in the study that produced a
+measurable improvement.
+
+## Evidence
+
+The area task asked models to compute a closed mesh's total area and say whether
+it matched the analytic value. Every interface computed the identical number,
+`12.566370614678554`, against `4*pi = 12.566370614359172`, a true error of
+`3.19e-10` and well inside the `1e-9` tolerance. The task was to *say so*.
+
+Without a check block, models got the arithmetic wrong in ways that are easy to
+reproduce:
+
+- All ten Gemma runs under both MCP variants reported an error of
+  `3.19e-11` — exactly ten times too small, contradicting the two numbers they
+  had just printed themselves.
+- GPT-5.5 sometimes omitted the tolerance or the verdict.
+- Four Gemma runs writing their own code summed flat triangles between mesh
+  vertices instead of spherical areas, returning `12.5590730782`, wrong by
+  `7.3e-3`.
+
+Adding a block stating the reference, residual, tolerance, and verdict took the
+task from **11/20 to 20/20**, Fisher exact `p = 1.2e-3`, surviving Bonferroni
+correction across all eight contrasts tested. Every deployment went to 5/5;
+Gemma went from 0/5 to 5/5.
+
+## Two honest caveats
+
+**The block is generous.** The prompt asked for five fields and the block
+supplied four of them outright, including the pass/fail verdict. A model can
+succeed by copying rather than by checking. So the demonstrated finding is that
+*doing the comparison on the server removes a class of arithmetic error models
+reliably make*, which is weaker than "models learned to verify" but is still a
+good reason to build it.
+
+**Under the strict scorer this result is not significant** (8/20 vs 13/20,
+`p = 0.20`). The seven disputed runs all report the five requested numbers
+correctly and are rejected only on JSON formatting, five of them Gemma, which
+emitted bare JSON in 0 of 120 runs. We think the lenient reading is right, but
+it should be known.
+
+## Suggested direction
+
+Two separable pieces.
+
+**The cheap one, worth doing regardless.** Every analysis result should carry a
+field that says whether a postcondition was evaluated, even when the answer is
+"no." An explicit `not_evaluated` costs almost nothing and stops a model
+implying more confidence than the computation supports. Four of the study's six
+tasks had no independent answer available, and saying so plainly is the right
+behavior.
+
+**The valuable one.** Where a genuine invariant exists, evaluate it server-side
+and return reference, residual, tolerance, and verdict. Candidates that already
+have known answers:
+
+- total area of a closed mesh equals `4*pi*R^2`
+- `curl(grad(phi))` is zero to discretization error
+- face areas sum to the global area
+- conservative remap preserves the area-weighted integral
+
+`scripts/analytic_validation.py` already computes residuals of this kind offline
+for manufactured solutions. It is not reachable from the server. Wiring that
+logic into the result path is most of the work.
+
+Be careful not to reintroduce issue 02: the check block that worked was **257
+bytes**. Keep it that size.

docs/issues-from-escience-study/04-remap-extrapolates-silently.md

@@ -0,0 +1,50 @@
+# Remapping returns confident numbers outside source coverage
+
+## Problem
+
+`remap_to_rectilinear` will happily return a value at every requested target
+point even when the target lies entirely outside the source mesh. There is no
+coverage check anywhere in the remap path — grepping `src/uxarray_mcp/` for
+`coverage` or `extrapolat` finds only unrelated matches in `plotting.py`.
+
+This is the failure mode a scientific interface most needs to catch, because
+nothing about the output looks wrong.
+
+## Evidence
+
+The study's remap fixture is deliberately adversarial. The source is a 195-cell
+regional mesh spanning roughly 43-47 degrees E and plus/minus 2 degrees
+latitude. The request asks for a global 5x5 lon/lat target.
+
+**Zero of the 25 target points fall inside the source mesh bounding box.**
+Nearest-neighbor remapping returned a value at all 25 anyway, with plausible
+statistics: min 0.138, max 0.159, mean 0.146. Every MCP run reproduced these
+numbers exactly and reported them as the answer.
+
+The study scored this task on invocation only and stated in print that the
+returned field is scientifically meaningless. That is a workaround for an
+evaluation, not an acceptable property of a server.
+
+A second issue compounds it: nearest-neighbor is **not conservative**, so even
+with full coverage the result would be unsuitable for any flux quantity. Nothing
+in the response says this.
+
+## Suggested direction
+
+At minimum, compute what fraction of target points fall within the source mesh
+and return it. A result carrying `points_in_source: 0/25` is impossible to
+misread, and it is cheap: a bounding-box test first, then a point-in-cell test
+only for points that pass.
+
+Beyond that:
+
+- Emit a stable warning code when coverage is partial, and a distinct one when
+  it is zero.
+- State the remap method's conservation property in the result, so a model
+  choosing between nearest-neighbor and a conservative scheme has the
+  information in front of it.
+- Consider returning masked or null values outside coverage rather than
+  extrapolated ones, or at minimum make that an option that defaults to safe.
+
+This connects to issue 05: coverage of zero is arguably a case where the server
+should refuse rather than warn.

docs/issues-from-escience-study/05-warnings-inform-but-never-block.md

@@ -0,0 +1,55 @@
+# Warnings inform but never block
+
+## Problem
+
+The server can detect that two arrays are unsuitable as physical vector
+components and then compute the curl of them anyway. In the study's vector task
+it returned `VECTOR_COMPONENTS_UNVERIFIED` and produced a number in the same
+response.
+
+A warning that does not stop the next step is advice. Stronger models took it;
+weaker ones did not.
+
+## Evidence
+
+The fixture gives two pairs of arrays that are **byte-identical**.
+`component_x` equals `vector_u` and `component_y` equals `vector_v`. Only the
+second pair carries `eastward_sea_water_velocity` and
+`northward_sea_water_velocity` standard names with `m s-1` units. A model
+reasoning from the numbers alone cannot distinguish them; it has to read the
+metadata and act on it.
+
+Outcomes were flat across every interface: 9/20 writing code, 10/20 with the
+named operation, 10/20 with the enriched result. Adding more warning text did
+not change behavior (`p = 1.0`). GPT-5.4 Nano and Gemma repeatedly called curl
+on the unlabeled pair, or omitted radius scaling, or both.
+
+Two observations follow. Detection is not the gap — the server already knows.
+And more description does not close it, which is consistent with issue 02.
+
+## Suggested direction
+
+Let the server declare preconditions that a client can enforce, and make
+violation refusable rather than merely reported.
+
+A workable shape:
+
+- Operations declare preconditions as data, not prose. For `curl` computing
+  physical vorticity: both components carry velocity-like units, direction
+  identity is resolvable from `standard_name` or `long_name`, and radius scaling
+  is enabled.
+- When a precondition fails, the default is to **refuse and return the reason**,
+  with an explicit override for a caller who genuinely wants the unphysical
+  number.
+- After a refusal, return only the repair actions that would make the call
+  valid, so a model has a bounded set of next steps instead of free rein.
+
+The same machinery covers the validation case. In the study, models correctly
+stopped after a failed validation, but they stopped because the prompt told them
+to, not because anything prevented them from continuing. All 80 runs did the
+right thing here, so this is not urgent — but the safety came from the prompt,
+which will not always be there.
+
+Note that `not_evaluated` from issue 03 and precondition failure are different
+states and should stay distinct: one is "we did not check," the other is "we
+checked and it fails."