Skip to content

docs(examples): record analysis runs in the model and report them in a document - #556

Merged
HuiJun merged 12 commits into
developfrom
docs/analysis-results-demo
Sep 24, 2026
Merged

HuiJun merged 12 commits into
developfrom
docs/analysis-results-demo

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Nothing in the tree showed how a run's results get into a generated document. This adds examples/analysis-results-demo/, a worked example in which analysis runs are recorded into the model as part usages on the bundled AnalysisRecords vocabulary — the same shape %record/-record-run emits — and a document tabulates the records.

private import AnalysisRecords::*;
part def DemoRun :> AnalysisRecords::AnalysisRun { attribute command : String; }
part def FuelBudgetRun :> DemoRun {
    ref part lander : Lander;
    attribute burnTime, fuelUsed, wetMass, fuelLeft : Real;      // as printed
    attribute liveFuelLeft : Real = lander.fuel - burnTime * lander.burnRate;
    attribute drift : Real = liveFuelLeft - fuelLeft;
    attribute stale : Boolean = drift != 0.0;
}
part scoutRun : FuelBudgetRun {
    @AnalysisRecords::RecordedRun { runAt = …; tool = …; command = …; kind = "run"; }
    ref :>> 'subject' = Landers::scout;  ref part :>> lander = scout;
    part verdict1 : VerdictRecord :> verdicts { … }
    …
}

The top-level Records package — -record-run's default target beside Descent — holds seven records (three baseline runs, a three-point sweep with iteration, one trade study whose scores are EvaluationRecord parts under evaluations) whose numbers are the tool's actual output; Reporting::AnalysisReport walks Descendants(Named("Records")), opens with an "Every recorded run" table filtered on AnalysisRecords::AnalysisRun, groups the fuel budgets by subject, isolates the sweep rows, lists provenance over WhereMetadata("AnalysisRecords::RecordedRun"), shows the trade study and its evaluations, and contrasts them with Verdicts recomputed at render time. relay.fuel was deliberately edited after recording, so exactly relayRun reports stale = true, drift = 30.0, and the README shows why the trade record silently disagrees instead (it rederives nothing).

The records are hand-written rather than produced by -record-run because the stale-relay story needs values from before the model edit and the derived drift/stale columns live on a shared def, while -record-run writes one def per case with no recompute. The README shows the -record-run command over this model and the record it generates, and the report rendered with -record-run "Descent::scoutBudget" shows the generated scoutBudget_run1 beside the hand-written rows in the all-runs and Provenance tables (the FuelBudgetRun-filtered tables exclude it, as documented).

Limitations the example documents: metadata annotation attributes are not projectable (command is duplicated as a plain attribute on DemoRun), the library's 'subject' ref is untyped so the drift formula reads a typed lander ref, objective is a reserved word, computed Column expressions do not support !=, and a part def matching a WhereType filter surfaces its own unbound features (the defs live in a nested Vocab package).

Also: a Walkthroughs row in examples/README.md, a paragraph in docs/manual/query-kinds.md pointing at the demo, a changelog fragment, and the re-recorded pilot-differential baseline and generated figures (the new file adds 397 pilot-only diagnostics, all on document-query calls and AnalysisRecords constructs the pinned pilot has no support for; adjudicated in docs/project/pilot-differential.md).

How it was verified

  • New TestAnalysisResultsExample (cmd/sysml): the render matches the committed report.md byte-for-byte, a render with -record-run "Descent::scoutBudget" lists scoutBudget_run1 in the all-runs and Provenance tables with the stale table unchanged, Reporting::StaleRuns returns exactly relayRun with drift = 30.0, and -analysis Descent::scoutBudget still prints the fuelLeft the scout record saved.
  • TestCorpusRoundTrip / API-JSON round trip: the new model is stable; the only expectation change is its added line.
  • go run -C tools ./cmd/pilot-diff -update, doc-counts, and go test ./referee/diff ./referee/xpect ./census/doccounts in tools.
  • go build ./... && go vet ./..., gofmt -l . empty, ./bin/sysml … -validate clean, scripts/check-doc-links.py, scripts/check-doc-ids.py, scripts/changelog.py check.

Checklist

  • make test and make lint pass locally
  • Tests added or updated for the change
  • Documentation extended where it already covers the surface (see CONTRIBUTING.md)
  • Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved (compliance rows need nothing: the census is counted at docs build)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/16d2adef11b1413e9656e2d67500f099
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/16d2adef11b1413e9656e2d67500f099?variant=devin
Requested by: @HuiJun

devin-ai-integration Bot and others added 2 commits September 24, 2026 01:29
…a document

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration Bot and others added 3 commits September 24, 2026 01:37
…demo

Co-Authored-By: jason.han <hanhuijun@gmail.com>
… example

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review September 24, 2026 03:27

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

devin-ai-integration Bot and others added 3 commits September 24, 2026 06:07
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…s library

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…sed demo

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 4 commits September 24, 2026 06:33
…ge so recorded runs join it

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…oted demo

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@HuiJun
HuiJun merged commit ee2e18e into develop Sep 24, 2026
15 checks passed
@HuiJun
HuiJun deleted the docs/analysis-results-demo branch September 24, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant