diff --git a/.agents/skills/testing-pilot-corpora-gate/SKILL.md b/.agents/skills/testing-pilot-corpora-gate/SKILL.md index 133c67d117..eeeeacca87 100644 --- a/.agents/skills/testing-pilot-corpora-gate/SKILL.md +++ b/.agents/skills/testing-pilot-corpora-gate/SKILL.md @@ -183,8 +183,8 @@ gate's own helpers are package-private but reusable (`pilotCorporaGate.files(t)` `actionlint`, `shellcheck`, `python3 scripts/check-doc-links.py`, `gofmt`, `go vet`, `go run -C tools ./cmd/pilot-diff` (validators pre-downloaded; ~4min, prints e.g. -the headline the committed baseline holds — `378 file(s), 347 fully agreeing; 38 agreed -diagnostic(s), 38 only ours, 1185 only the pilot's` after the Legend of the Red Dragon example left for its own repository at the `2026-08` pin, so read it from +the headline the committed baseline holds — `379 file(s), 347 fully agreeing; 38 agreed +diagnostic(s), 38 only ours, 1582 only the pilot's` at the `2026-08` pin, so read it from `docs/project/pilot-differential-baseline.json` rather than from this line) and `make lint` (staticcheck+gosec, ~2min) all work. There is **no** `yamllint` and **no** `circleci` CLI, so `.circleci/config.yml` can only be parsed as YAML, not schema-validated — say so diff --git a/.agents/skills/testing-pilot-differential/SKILL.md b/.agents/skills/testing-pilot-differential/SKILL.md index ab2579db15..bea4275b73 100644 --- a/.agents/skills/testing-pilot-differential/SKILL.md +++ b/.agents/skills/testing-pilot-differential/SKILL.md @@ -21,9 +21,9 @@ GNU-format diagnostics **relative to `--root`**. Consequences for testing: - The pin `tools/referee/diff` reports comes from `build/pilot-sysml-validator/pilot-pin.txt` (written by the new script), not from the DeciSym `pom.xml`. - `-validator /nonexistent` now says `run ./scripts/download-pilot-sysml-validator.sh`. -- Measured after the Legend of the Red Dragon example left for its own repository at the `2026-08` pin, with a fresh library cache: `378 file(s), 347 fully agreeing; 38 agreed, - 38 only ours, 1185 only the pilot's`, JSON totals `openSysMLDiagnostics 79 / pilotDiagnostics - 1226 / severityMismatch 3`; ~2 min wall, byte-identical across runs *and* after a from-scratch +- Measured at the `2026-08` pin, with a fresh library cache: `379 file(s), 347 fully agreeing; 38 agreed, + 38 only ours, 1582 only the pilot's`, JSON totals `openSysMLDiagnostics 79 / pilotDiagnostics + 1623 / severityMismatch 3`; ~2 min wall, byte-identical across runs *and* after a from-scratch rebuild of `build/pilot-validator`. The six `kerml-examples` pilot-only rows the `2026-07` run carried (`The opposite features 'owningType' … do not refer to each other`) are gone: the pilot fixed its `ownedDisjoining` delegate, and nothing on our side moved. `kerml-examples` carries no `syntax` diagnostic on either @@ -137,7 +137,7 @@ The harness compares OpenSysML diagnostics against the OMG SysML v2 Pilot Implem `build/pilot-diff/pilot-diff.{txt,json}`. `docs/project/pilot-differential-baseline.json` is the committed result of the *last refreshed* run, so **the harness is testable by reproduction** — but only while the baseline is current. Check that first. As of the rebaseline that came when the Legend of the Red Dragon example left for its own repository it **is** -current: a live run gives `378 file(s), 347 fully agreeing; 38 agreed, 38 only ours, 1185 only the +current: a live run gives `379 file(s), 347 fully agreeing; 38 agreed, 38 only ours, 1582 only the pilot's`, byte-identical to the committed baseline, and `docs/project/pilot-differential.md`'s "Results" table matches. The rebaseline before it, at the architecture self-model's landing, covered two rounds, because the succession-shorthand removal before it landed without refreshing the baseline; a control run of its merge commit gives diff --git a/.agents/skills/testing-pilot-execution-referee/SKILL.md b/.agents/skills/testing-pilot-execution-referee/SKILL.md index 0696b77dd5..01e3443194 100644 --- a/.agents/skills/testing-pilot-execution-referee/SKILL.md +++ b/.agents/skills/testing-pilot-execution-referee/SKILL.md @@ -148,8 +148,8 @@ pilot answers the representation's own. See `pilot-exec-diff: :: model no/such/model.sysml: stat : no such file or directory`. - **Additivity.** `go run -C tools ./cmd/pilot-diff` must still print the headline the - committed baseline holds (`378 file(s), 347 fully agreeing; 38 agreed - diagnostic(s), 38 only ours, 1185 only the pilot's` after the Legend of the Red Dragon example left for its own repository at the `2026-08` pin — read it from the baseline JSON, not from this line, since each + committed baseline holds (`379 file(s), 347 fully agreeing; 38 agreed + diagnostic(s), 38 only ours, 1582 only the pilot's` at the `2026-08` pin — read it from the baseline JSON, not from this line, since each fix round moves it) and `jq -S` diff clean against `docs/project/pilot-differential-baseline.json`; `git status --porcelain` empty at the end. diff --git a/.agents/skills/testing-pilot-xpect/SKILL.md b/.agents/skills/testing-pilot-xpect/SKILL.md index 0f31b8c8b9..e36f10d4e1 100644 --- a/.agents/skills/testing-pilot-xpect/SKILL.md +++ b/.agents/skills/testing-pilot-xpect/SKILL.md @@ -416,8 +416,8 @@ census in `w5c_census_test.go` is live two ways: perturb one pinned triple (e.g. ## Regression neighbour `go run -C tools ./cmd/pilot-diff` (~1m12s) must still print the headline the *committed* baseline holds — -after the Legend of the Red Dragon example left for its own repository at the `2026-08` pin that is `378 file(s), 347 fully agreeing; 38 agreed diagnostic(s), 38 -only ours, 1185 only the pilot's`. Read the number out of +at the `2026-08` pin that is `379 file(s), 347 fully agreeing; 38 agreed diagnostic(s), 38 +only ours, 1582 only the pilot's`. Read the number out of `docs/project/pilot-differential-baseline.json` rather than trusting this line, since a landing fix round moves it. When the baseline is itself stale (it was at `19a3ce03`, holding 273 / 281 / 317), a failing `cmp` against it is *not* evidence of an Xpect regression — compare the summary line, and see diff --git a/README.md b/README.md index 9caf7cee17..495c1e8288 100644 --- a/README.md +++ b/README.md @@ -313,11 +313,11 @@ The project is under active development, with the core infrastructure operationa **Measured against the pinned reference** (`PILOT_TAG=2026-08`, artifact `0.62.0`). Every number below is generated by `make docs-counts` from the committed baselines and gated; none of them is typed in by hand. -- **Corpus agreement:** 347 of 378 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1185 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](docs/project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). +- **Corpus agreement:** 347 of 379 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1582 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](docs/project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). - **Declared-diagnostic silence:** of the 512 declared `errors` rows in the reference's own Xpect suites, we report nothing for 0. 245 we report word-for-word; 248 wording-only and 7 location-only differences are agreement in substance and are not counted as gaps; 0 more we report as a warning and 2 elsewhere in the file ([Xpect oracle](docs/project/pilot-xpect.md), `go run -C tools ./cmd/pilot-xpect`). - **Scope agreement:** 230 of 230 declared scope assertions match exactly (same source). - **Permissiveness gaps:** of 306 invalid models we wrote ourselves, the reference rejects 4 that we accept by default, and 293 both reject; 4 further cases agree only when we are asked strictly. We authored every one of these cases ourselves, so the denominator measures the reach of our own corpus and not our conformance; agreement reached only under an opt-in strict mode is weaker evidence than agreement by default ([rejection oracle](docs/project/pilot-rejection.md), `go run -C tools ./cmd/pilot-reject`). -- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](docs/project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 378 files agreeing, 37 diagnostics ours alone and 1185 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. +- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](docs/project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 379 files agreeing, 37 diagnostics ours alone and 1582 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. - **Self-assessed surface:** the action, state-machine and classifier-behavior rows have no external referee at all — the four refereed figures above cannot see them, because the pinned artifact evaluates expressions but executes neither actions nor state machines. [Spec compliance](docs/project/spec-compliance.md) counts them. What these numbers cannot show: the OMG corpora are demonstrations rather than an official conformance suite; the differential is one-directional, comparing the diagnostics the two implementations report on the same files; the Xpect suites are the pilot authors' test intent rather than a certification oracle; and none of these is a percentage of the specification — no global compliance figure is claimed anywhere. @@ -329,7 +329,7 @@ What these numbers cannot show: the OMG corpora are demonstrations rather than a **Test coverage:** top-level `Test` functions (counted from the `_test.go` files, as `go test ./...` runs them) covering parsers, semantics, runtime (actions, states, instances, operators, validation), behind golden ASTs, negatives, execution conformance cases, golden traces, runtime robustness cases and gRPC conformance and robustness cases. The figures are counted from the tree when the documentation site is built into the test inventory of [spec compliance](docs/project/spec-compliance.md), never committed, so a branch adding a test does not rewrite this page. A test skips only for want of something the run did not provide, and says what: the held-image round trip declines a conformance case that creates no instance, a few gate on a PDF or Mermaid toolchain, a pinned pilot artifact, the PSSM suite, a locale, a case-insensitive filesystem or a live Flexo stack, and the OMG corpus gates skip until the corpora are downloaded unless asked to fail. **Parser coverage:** 105/105 bundled library files parse cleanly — the 94 official SysML v2 standard library files and the non-normative `OpenSysML Libraries/OpenSysMLMathFunctions.kerml`, `OpenSysML Libraries/DocumentQueries.sysml`, `OpenSysML Libraries/IdentityMetadata.sysml`, `OpenSysML Libraries/DiagramLayout.sysml`, `OpenSysML Libraries/OOSEM.sysml`, `OpenSysML Libraries/MOSA.sysml`, `OpenSysML Libraries/StateSpaceIntegration.sysml`, `OpenSysML Libraries/Stochastic.sysml`, `OpenSysML Libraries/RandomFunctions.kerml`, `OpenSysML Libraries/Simulation.sysml` and `OpenSysML Libraries/MigrationMetadata.sysml` extensions. Conformance verified by [stdlib_conformance_test.go](internal/workspace/libs/stdlib_conformance_test.go). Grammar reference: [OMG Xtext grammar](https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation/tree/master/org.omg.kerml.xtext/src/org/omg/kerml/xtext). **Behavioral execution:** Calc/constraint/requirement/satisfy functional. Action/state executors handle nested invocation, control flow keywords, loop and conditional statements and the send statement (every conformance case passing). Coverage is self-assessed against the specification text and the normative library: the pinned OMG pilot implementation evaluates expressions but does not execute actions or state machines headlessly, so no external implementation currently adjudicates these rows. See [spec compliance](docs/project/spec-compliance.md). -**Reference differential:** 378 files compared diagnostic-by-diagnostic against the pinned OMG pilot implementation (`2026-08`), 347 in full agreement; every divergence is enumerated and adjudicated in [the differential](docs/project/pilot-differential.md), reproducible with `go run -C tools ./cmd/pilot-diff`. +**Reference differential:** 379 files compared diagnostic-by-diagnostic against the pinned OMG pilot implementation (`2026-08`), 347 in full agreement; every divergence is enumerated and adjudicated in [the differential](docs/project/pilot-differential.md), reproducible with `go run -C tools ./cmd/pilot-diff`. **Rejection oracle:** the reverse direction — do we reject what the reference rejects? 306 hand-written invalid models validated by both implementations, 297 rejected by both, 0 the pinned pilot rejects and we accept; the remainder only we reject — the control-node succession rules the pinned pilot leaves unimplemented and a non-Boolean succession guard it accepts once the standard library types it — and every permissiveness gap is enumerated with a reproducer and likely root cause in [the rejection oracle](docs/project/pilot-rejection.md), reproducible with `go run -C tools ./cmd/pilot-reject`. We wrote every case, so the count measures our coverage of the rejection surface, not our conformance — a sample, not a proof. **Training examples:** 100/100 files clean, gated by `tests/corpus/testdata/training_examples_expected.txt`. Download with `./scripts/download-training-examples.sh` (from the [OMG training directory](https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation/tree/master/sysml/src/training)). See [training examples](docs/project/training-examples.md) for analysis. **Semantic layer:** a complete implementation of runtime operators, feature chains and validation rules. See [examples/semantic-layer/](examples/semantic-layer/) for a full demonstration. diff --git a/changes/unreleased/analysis-results-demo.added.md b/changes/unreleased/analysis-results-demo.added.md new file mode 100644 index 0000000000..9c14ad6347 --- /dev/null +++ b/changes/unreleased/analysis-results-demo.added.md @@ -0,0 +1 @@ +- **Added `examples/analysis-results-demo/`, a worked example of saving analysis runs into the model and reporting them in a document.** The records are `part` usages on the bundled `AnalysisRecords` vocabulary — the same shape `-record-run`/`%record` emits, written by hand to keep pre-edit values — and a generated report groups, filters and lists them, flags the record a later model edit made stale through a derived `drift`/`stale` pair, and contrasts them with `Verdicts` recomputed live at render time. diff --git a/cmd/sysml/examples_analysis_results_test.go b/cmd/sysml/examples_analysis_results_test.go new file mode 100644 index 0000000000..78e2c0e653 --- /dev/null +++ b/cmd/sysml/examples_analysis_results_test.go @@ -0,0 +1,91 @@ +package main + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// TestAnalysisResultsExample runs the analysis-results demo end to end: the +// rendered report matches its committed output, the stale-records query +// returns exactly the record the model moved away from, and the analysis +// still prints the fuelLeft the scout record saved. +func TestAnalysisResultsExample(t *testing.T) { + binary := buildCLI(t) + examples := filepath.Join("..", "..", "examples", "analysis-results-demo") + source := filepath.Join(examples, "lander-results.sysml") + + committed, err := os.ReadFile(filepath.Join(examples, "report.md")) + if err != nil { + t.Fatal(err) + } + out := filepath.Join(t.TempDir(), "report.md") + render := exec.Command(binary, source, "-render-document", "Reporting::AnalysisReport", "-o", out) + if output, err := render.CombinedOutput(); err != nil { + t.Fatalf("render: %v\n%s", err, output) + } + written, err := os.ReadFile(out) + if err != nil { + t.Fatal(err) + } + if string(written) != string(committed) { + t.Errorf("rendered example differs from examples/analysis-results-demo/report.md:\n%s", written) + } + + stale := exec.Command(binary, source, "-run-query", "Reporting::StaleRuns") + output, err := stale.CombinedOutput() + if err != nil { + t.Fatalf("stale query: %v\n%s", err, output) + } + for _, want := range []string{ + "returned 1 row", + "Row 1: Records::relayRun", + "drift = 30.0", + "liveFuelLeft = 110.0", + } { + if !strings.Contains(string(output), want) { + t.Errorf("stale query output is missing %q:\n%s", want, output) + } + } + + // The recorded baseline must match what the case still prints; if the + // model moves so the printed value differs from scoutRun's fuelLeft, this + // catches the record silently drifting. + run := exec.Command(binary, source, "-analysis", "Descent::scoutBudget") + output, err = run.CombinedOutput() + if err != nil { + t.Fatalf("analysis: %v\n%s", err, output) + } + if !strings.Contains(string(output), "fuelLeft = 130.0") { + t.Errorf("scoutBudget no longer prints the recorded fuelLeft = 130.0:\n%s", output) + } + + // A run written back with -record-run lands in the same Records package + // the report's queries walk, so the document must show it alongside the + // hand-written records — and only it, in the run-table filters that name + // the hand-written defs. + t.Run("RecordedRunJoinsTheReport", func(t *testing.T) { + recorded := filepath.Join(t.TempDir(), "recorded-report.md") + render = exec.Command(binary, source, "-record-run", "Descent::scoutBudget", + "-render-document", "Reporting::AnalysisReport", "-o", recorded) + if output, err := render.CombinedOutput(); err != nil { + t.Fatalf("render with -record-run: %v\n%s", err, output) + } + written, err := os.ReadFile(recorded) + if err != nil { + t.Fatal(err) + } + text := string(written) + if !strings.Contains(text, "| scoutBudget\\_run1 | Descent::scoutBudget | run | Landers::scout | satisfied |") { + t.Errorf("recorded run missing from the every-recorded-run table:\n%s", text) + } + if !strings.Contains(text, "| scoutBudget\\_run1 | run | Descent::scoutBudget |") { + t.Errorf("recorded run missing from the provenance table:\n%s", text) + } + if strings.Count(text, "| relayRun | Landers::relay | 40 | 80 | 110 | 30 |") != 1 { + t.Errorf("stale table does not contain exactly the relay record:\n%s", text) + } + }) +} diff --git a/docs/internals/architecture.md b/docs/internals/architecture.md index 6b78b70c11..8006044f84 100644 --- a/docs/internals/architecture.md +++ b/docs/internals/architecture.md @@ -840,11 +840,11 @@ Every behavioral feature must have: **Measured against the pinned reference** (`PILOT_TAG=2026-08`, artifact `0.62.0`). Every number below is generated by `make docs-counts` from the committed baselines and gated; none of them is typed in by hand. -- **Corpus agreement:** 347 of 378 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1185 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](../project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). +- **Corpus agreement:** 347 of 379 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1582 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](../project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). - **Declared-diagnostic silence:** of the 512 declared `errors` rows in the reference's own Xpect suites, we report nothing for 0. 245 we report word-for-word; 248 wording-only and 7 location-only differences are agreement in substance and are not counted as gaps; 0 more we report as a warning and 2 elsewhere in the file ([Xpect oracle](../project/pilot-xpect.md), `go run -C tools ./cmd/pilot-xpect`). - **Scope agreement:** 230 of 230 declared scope assertions match exactly (same source). - **Permissiveness gaps:** of 306 invalid models we wrote ourselves, the reference rejects 4 that we accept by default, and 293 both reject; 4 further cases agree only when we are asked strictly. We authored every one of these cases ourselves, so the denominator measures the reach of our own corpus and not our conformance; agreement reached only under an opt-in strict mode is weaker evidence than agreement by default ([rejection oracle](../project/pilot-rejection.md), `go run -C tools ./cmd/pilot-reject`). -- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](../project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 378 files agreeing, 37 diagnostics ours alone and 1185 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. +- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](../project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 379 files agreeing, 37 diagnostics ours alone and 1582 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. - **Self-assessed surface:** the action, state-machine and classifier-behavior rows have no external referee at all — the four refereed figures above cannot see them, because the pinned artifact evaluates expressions but executes neither actions nor state machines. [Spec compliance](../project/spec-compliance.md) counts them. What these numbers cannot show: the OMG corpora are demonstrations rather than an official conformance suite; the differential is one-directional, comparing the diagnostics the two implementations report on the same files; the Xpect suites are the pilot authors' test intent rather than a certification oracle; and none of these is a percentage of the specification — no global compliance figure is claimed anywhere. diff --git a/docs/manual/query-kinds.md b/docs/manual/query-kinds.md index b8ea0920f0..1bc65ef53d 100644 --- a/docs/manual/query-kinds.md +++ b/docs/manual/query-kinds.md @@ -80,7 +80,12 @@ record is written into the model as elements annotated `@AnalysisRecords::RecordedRun`, so `WhereMetadata` finds each one and `WhereFeature`/`Project`/`OrderBy` read the values it bound — `caseName`, `kind`, `iteration`, and a property per input and output. See -[Recording analysis runs](recording-analysis-runs.md). +[Recording analysis runs](recording-analysis-runs.md). A plain `-analysis`, +`-sweep` or trade study still prints and discards its results, so a run that +was not recorded leaves nothing a query can see; +[the analysis-results demo](../../examples/analysis-results-demo/README.md) +tables records in the same vocabulary, including records that flag themselves +stale when the model moves. ## Runtime state and event queries diff --git a/docs/project/pilot-differential-baseline.json b/docs/project/pilot-differential-baseline.json index 7b76b7f2ae..d005f55e47 100644 --- a/docs/project/pilot-differential-baseline.json +++ b/docs/project/pilot-differential-baseline.json @@ -60,8 +60,8 @@ "name": "examples", "dir": "examples", "origin": "ours", - "files": 43, - "digest": "sha256:3e4522077d7c4903aa3c6cad01dc84abc79588f9f13c62e7c5aee8c9bee1a02f" + "files": 44, + "digest": "sha256:e1102754c82d55de49dd9a70010c43e515784f8237de200fc7e29d79ff8a7bcd" }, { "name": "probes", @@ -74,14 +74,14 @@ "recorded": "2026-09-24" }, "totals": { - "files": 378, + "files": 379, "filesFullyAgreeing": 347, "agreement": 38, "severityMismatch": 3, "openSysMLOnly": 38, - "pilotOnly": 1185, + "pilotOnly": 1582, "openSysMLDiagnostics": 79, - "pilotDiagnostics": 1226 + "pilotDiagnostics": 1623 }, "roots": [ { @@ -746,16 +746,1758 @@ "name": "examples", "dir": "examples", "totals": { - "files": 43, + "files": 44, "filesFullyAgreeing": 30, "agreement": 4, "severityMismatch": 2, "openSysMLOnly": 7, - "pilotOnly": 1165, + "pilotOnly": 1562, "openSysMLDiagnostics": 13, - "pilotDiagnostics": 1171 + "pilotDiagnostics": 1568 }, "files": [ + { + "path": "analysis-results-demo/lander-results.sysml", + "agreement": [], + "severityMismatch": [], + "openSysMLOnly": [], + "pilotOnly": [ + { + "line": 164, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 174, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 205, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 205, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 206, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 206, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 207, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 207, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 208, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 208, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 209, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 209, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 211, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 212, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 213, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 215, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 217, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 222, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 222, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 223, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 224, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 225, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 229, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 229, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 230, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 230, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 231, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 231, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 232, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 232, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 233, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 233, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 235, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 236, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 237, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 239, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 241, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 246, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 246, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 247, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 248, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 249, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 253, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 253, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 254, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 254, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 255, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 255, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 256, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 256, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 257, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 257, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 259, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 260, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 261, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 263, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 265, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 270, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 270, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 271, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 272, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 273, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 279, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 279, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 280, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 280, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 281, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 281, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 282, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 282, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 283, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 283, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 285, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 286, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 287, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 289, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 291, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 292, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 297, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 297, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 298, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 299, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 300, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 304, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 304, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 305, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 305, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 306, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 306, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 307, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 307, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 308, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 308, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 310, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 311, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 312, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 314, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 316, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 317, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 322, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 322, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 323, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 324, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 325, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 329, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 329, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 330, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 330, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 331, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 331, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 332, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 332, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 333, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 333, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 335, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 336, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 337, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 339, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 341, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 342, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 347, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 347, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 348, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 349, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 350, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 358, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 358, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 359, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 359, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 360, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 360, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 361, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 361, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 362, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 362, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 364, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 365, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 366, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 369, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 369, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 370, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 371, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 372, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 374, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 374, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 375, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 376, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 377, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 378, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 379, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 379, + "severity": "warning", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 380, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 382, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 382, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 383, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 384, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 385, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 386, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 387, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 387, + "severity": "warning", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 388, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 390, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 390, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 391, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 392, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 393, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 394, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 395, + "severity": "error", + "category": "unmapped", + "count": 1 + }, + { + "line": 395, + "severity": "warning", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 396, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 403, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 408, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 409, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 409, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 410, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 410, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 411, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 411, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 412, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 412, + "severity": "error", + "category": "unresolved-reference", + "count": 6 + }, + { + "line": 413, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 414, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 415, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 419, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 420, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 420, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 421, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 421, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 422, + "severity": "error", + "category": "kind-mismatch", + "count": 3 + }, + { + "line": 422, + "severity": "error", + "category": "unresolved-reference", + "count": 9 + }, + { + "line": 423, + "severity": "error", + "category": "unresolved-reference", + "count": 4 + }, + { + "line": 424, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 428, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 429, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 429, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 430, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 430, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 431, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 431, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 432, + "severity": "error", + "category": "kind-mismatch", + "count": 3 + }, + { + "line": 432, + "severity": "error", + "category": "unresolved-reference", + "count": 9 + }, + { + "line": 433, + "severity": "error", + "category": "unresolved-reference", + "count": 3 + }, + { + "line": 434, + "severity": "error", + "category": "unresolved-reference", + "count": 4 + }, + { + "line": 435, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 439, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 440, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 440, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 441, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 441, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 442, + "severity": "error", + "category": "kind-mismatch", + "count": 3 + }, + { + "line": 442, + "severity": "error", + "category": "unresolved-reference", + "count": 9 + }, + { + "line": 443, + "severity": "error", + "category": "unresolved-reference", + "count": 3 + }, + { + "line": 444, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 448, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 449, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 449, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 450, + "severity": "error", + "category": "kind-mismatch", + "count": 3 + }, + { + "line": 450, + "severity": "error", + "category": "unresolved-reference", + "count": 9 + }, + { + "line": 451, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 455, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 456, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 456, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 457, + "severity": "error", + "category": "kind-mismatch", + "count": 3 + }, + { + "line": 457, + "severity": "error", + "category": "unresolved-reference", + "count": 9 + }, + { + "line": 458, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 462, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 463, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 463, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 464, + "severity": "error", + "category": "kind-mismatch", + "count": 3 + }, + { + "line": 464, + "severity": "error", + "category": "unresolved-reference", + "count": 9 + }, + { + "line": 465, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 469, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 471, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 471, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 472, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 472, + "severity": "error", + "category": "unresolved-reference", + "count": 3 + }, + { + "line": 473, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 477, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 478, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 480, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 480, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 481, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 484, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 484, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 485, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 486, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 486, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 487, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 492, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 492, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 493, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 494, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 494, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 495, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 496, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 501, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 501, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 502, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 503, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 503, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 504, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 509, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 509, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 510, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 511, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 511, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 512, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 517, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 517, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 518, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 519, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 519, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 520, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 525, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 525, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 526, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 527, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 527, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 528, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 531, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 531, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 532, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 537, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 537, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 538, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 539, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 539, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 540, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 542, + "severity": "warning", + "category": "kind-mismatch", + "count": 1 + } + ] + }, { "path": "disposal-robot-demo/robot.sysml", "agreement": [], @@ -6086,6 +7828,11 @@ } ], "unmapped": [ + { + "side": "pilot", + "message": "Cannot override a binding feature value", + "count": 10 + }, { "side": "pilot", "message": "Duplicate of other owned member name", @@ -6436,14 +8183,14 @@ } ], "totals": { - "files": 378, + "files": 379, "filesFullyAgreeing": 348, "agreement": 38, "severityMismatch": 3, "openSysMLOnly": 37, - "pilotOnly": 1185, + "pilotOnly": 1582, "openSysMLDiagnostics": 78, - "pilotDiagnostics": 1226 + "pilotDiagnostics": 1623 }, "findings": [ { diff --git a/docs/project/pilot-differential.md b/docs/project/pilot-differential.md index eeea64846d..a7c7d5fdcf 100644 --- a/docs/project/pilot-differential.md +++ b/docs/project/pilot-differential.md @@ -209,7 +209,7 @@ nor double-counted as two independent disagreements. --- -## Results (pilot `2026-08`, 378 files) +## Results (pilot `2026-08`, 379 files) | Root | Files | Fully agreeing | Ours | Pilot | Agreed | Severity-only | Only ours | Only pilot | |---|---:|---:|---:|---:|---:|---:|---:|---:| @@ -218,9 +218,9 @@ nor double-counted as two independent disagreements. | `examples/pilot-corpora/sysml-validation` | 56 | 56 | 0 | 0 | 0 | 0 | 0 | 0 | | `examples/pilot-corpora/kerml-examples` | 58 | 55 | 10 | 0 | 0 | 0 | 10 | 0 | | `tests/testdata` | 18 | 10 | 43 | 55 | 34 | 1 | 8 | 20 | -| `examples` | 43 | 30 | 13 | 1171 | 4 | 2 | 7 | 1165 | +| `examples` | 44 | 30 | 13 | 1568 | 4 | 2 | 7 | 1562 | | `tools/referee/diff/testdata` (probes) | 4 | 1 | 6 | 0 | 0 | 0 | 6 | 0 | -| **Total** | **378** | **347** | **79** | **1226** | **38** | **3** | **38** | **1185** | +| **Total** | **379** | **347** | **79** | **1623** | **38** | **3** | **38** | **1582** | **Read the `only ours` total by root, never as one number.** Step 2 removes nine resolver false positives from the reference's **own** corpora: `pilot-examples` 16 → **7** and @@ -791,8 +791,8 @@ cascades through the rest of the file. The movement is entirely one file, | Count | Before the initializer rewrite | Now | |---|---:|---:| -| only pilot | 82 | **1185** | -| pilot diagnostics | 123 | **1226** | +| only pilot | 82 | **1582** | +| pilot diagnostics | 123 | **1623** | | severity-only | 9 | **3** | The rewrite itself took only-pilot to 61 and pilot diagnostics to 101; the `Now` column states @@ -925,7 +925,7 @@ Per category, the only-ours totals are: `pilot-examples` 4 `unmapped`, 2 advisory of the [runtime showcase round](#runtime-showcase-round)); `testdata` 7 `unmapped`, 1 `multiplicity`; `probes` 6 `unmapped`. Only-pilot: `testdata` 12 `kind-mismatch`, 3 `unmapped`, 3 syntax, 2 `unresolved-reference`; -`examples` 10 syntax, 19 `unmapped`, 547 `kind-mismatch`, 589 `unresolved-reference` — of which +`examples` 10 syntax, 29 `unmapped`, 661 `kind-mismatch`, 862 `unresolved-reference` — of which `relay-probe-demo/mission.sysml` carries none: it carried a `kind-mismatch` on its send of a `Telemetry` invocation until the send-argument round above, and the demo now writes the constructor, `send new Telemetry(…) via antenna`, which both implementations accept, so the row @@ -1017,13 +1017,13 @@ page's history. | Count | Now | |---|---:| | overall: fully agreeing / only ours / our diagnostics | **347 / 38 / 79** | -| only pilot | **1185** | -| pilot diagnostics | **1226** | +| only pilot | **1582** | +| pilot diagnostics | **1623** | | severity-only | **3** | | unmapped, our side | **34** | | kerml-examples: only ours | **10** | | pilot-examples: only ours | **7** | -| examples: only pilot | **1165** | +| examples: only pilot | **1562** | The KerML root is now the *cleanest* of the three OMG roots in proportion: **10** only-ours against 6 only-pilot, with 49 of 58 files fully @@ -1168,6 +1168,20 @@ of them the `kind-mismatch` rows adjudicated below where a `calc def` is passed Nothing else moves: the file draws no diagnostic from this implementation, so `fully agreeing`, `only ours` and `agreed` stay where the analysis walkthrough left them. +### Analysis results recording round + +`examples/analysis-results-demo/lander-results.sysml` is one file added to the `examples` root: +files 43 → **44** on the root, 378 → **379** overall, and pilot diagnostics 1226 → **1623** / +only pilot 1185 → **1582** — 397 diagnostics in 289 reported rows: 192 `unresolved-reference` +(counted 273), 87 `kind-mismatch` (counted 114) and 10 `unmapped`, all inside the file's +`Records` and `Reporting` packages. Every row is a construct the pinned artifact has no support +for: the document-query calls (`Project`, `OrderBy`, `WhereType`, `WhereFeature`, +`WhereMetadata`, `Verdicts`, `Descendants`), the `@AnalysisRecords::RecordedRun` metadata +annotations and specializations of the `AnalysisRecords` library defs, and the run-record part +usages' quoted `'objective'`/`'subject'` names and `ref part :>>`/`part :>>` redefinitions. The +file draws no diagnostic from this implementation — it validates clean — so `fully agreeing`, +`only ours`, `agreed` and `severity-only` all stay where the expressions walkthrough left them. + ## Adjudications ### Only ours — candidate false positives (3, SysML side) diff --git a/examples/README.md b/examples/README.md index ef7e5fb107..5b58c05548 100644 --- a/examples/README.md +++ b/examples/README.md @@ -32,6 +32,7 @@ Each of these is a model and a walkthrough of the commands that exercise it. | [relay-probe-demo/mission.sysml](relay-probe-demo/mission.sysml) | [relay-probe-demo/README.md](relay-probe-demo/README.md) | one individual probe across its mission phases: event occurrences ordered in time, snapshots and a timeslice of one individual, occurrences with multiplicity, a calculation reading across two snapshots, a requirement whose subject is a snapshot, and a beacon inside a timeslice sending telemetry through its probe's own port | | [analysis-demo/lander.sysml](analysis-demo/lander.sysml) | [analysis-demo/README.md](analysis-demo/README.md) | analysis cases, asked every way the tool answers them: an analysis whose action steps feed each other and whose objective is a requirement, run bound, with arguments and on an object; a verification case whose body decides its verdict beside its objective; a parameter sweep and a seeded sample; two trade studies choosing among three landers; an action and a state machine due at the same instant of one clock, under each scheduling policy and explored; `-trace`, `-json`, the REPL forms and [the same questions from Python](analysis-demo/lander_demo.py) | | [verdicts-demo/rover.sysml](verdicts-demo/rover.sysml) | [verdicts-demo/README.md](verdicts-demo/README.md) | `Verdicts(...)` in a document query: one row per assertion about a rover and every object it holds — its constraints, a requirement it carries, a `satisfy` of its battery with the verification case verifying it, six wheels from a multiplicity and a heater no value decides — over the declared object, then over the object a session holds after a drive; what the table sees that evaluating one expression does not | +| [analysis-results-demo/lander-results.sysml](analysis-results-demo/lander-results.sysml) | [analysis-results-demo/README.md](analysis-results-demo/README.md) | saving analysis runs into the model: `-analysis` and `-sweep` print and discard, so each run is written back as a `part` usage typed by a result-record definition with `@RecordedRun` provenance metadata and a `ref part` to its subject — then a document groups the records by subject, filters the sweep rows and the trade study, lists every annotated record's command, and flags the record a later model edit made stale, beside live `Verdicts` recomputed at render time | | [solver-demo.sysml](solver-demo.sysml) | [SOLVER-DEMO.md](SOLVER-DEMO.md) | `%check`, `%explain`, `%solve`, `%configure` and `%optimize` — what conditions *can* hold, which conflict, what satisfies them, which variants are permitted, what is best (needs z3 or cvc5) | | [oosem-demo/oosem-demo.sysml](oosem-demo/oosem-demo.sysml) | [oosem-demo/README.md](oosem-demo/README.md) | the `OOSEM` library on a small Earth-observation mission: as-is and to-be enterprise, causal analysis, stakeholder needs derived down to component requirements with `#moe`/`#mop`, the black-box system context and its use case, the logical scenario and components, and the physical architecture distributed over nodes | | [mosa-demo/mosa-demo.sysml](mosa-demo/mosa-demo.sysml) | [mosa-demo/README.md](mosa-demo/README.md) | the `MOSA` library on a modular ground vehicle: the major system platform, its major system components and a modular autonomy system, the modular system interfaces between them (one written as `#keyInterface`), the consensus standards they conform to, data rights and proprietary elements, interface control, MOSA requirements with their traces, a conformance assessment, the MOSA views and a generated interface control document; `-validate` reports the openness gaps the model leaves on purpose | diff --git a/examples/analysis-results-demo/README.md b/examples/analysis-results-demo/README.md new file mode 100644 index 0000000000..9c29b43b2f --- /dev/null +++ b/examples/analysis-results-demo/README.md @@ -0,0 +1,346 @@ +# Analysis results demo: saving runs into the model and reporting them + +[`lander-results.sysml`](lander-results.sysml) asks one question: **can the +results of analysis runs be saved into the model itself, so a generated +document can tabulate them later?** + +The honest answer is *yes — by recording them, which the tool can now do +for you*. `-analysis` and `-sweep` print each run's inputs, outputs and +verdicts, but a run leaves nothing a document query can read until it is +**recorded**: `%record` and `-record-run` write the run back into the model as +a `part` usage typed by a result-record definition in the bundled +`AnalysisRecords` library — inputs, outputs, objective status, verdicts and +evaluations, annotated with provenance metadata and a `ref` to the part the +run was about. This demo's records are the same shape, written by hand (why, +below). [`report.md`](report.md) is what the document then renders. + +## The recording pattern + +The `Records` package builds the demo's record definitions on the bundled +`AnalysisRecords` library, which supplies `RecordedRun` (provenance +metadata: `runAt`, `tool`, `command`, `kind`), `AnalysisRun` (`caseName`, +`kind`, `'objective'`, `iteration`, `'subject'`, `subjectName`, `verdict`, +plus `verdicts` and `evaluations` collections), `VerdictRecord` and +`EvaluationRecord`: + +```sysml +part def DemoRun :> AnalysisRecords::AnalysisRun { + attribute command : String; +} + +part def FuelBudgetRun :> DemoRun { + ref part lander : Lander; + attribute burnTime : Real; + attribute fuelUsed : Real; + attribute wetMass : Real; + attribute fuelLeft : Real; + attribute liveFuelLeft : Real = lander.fuel - burnTime * lander.burnRate; + attribute drift : Real = liveFuelLeft - fuelLeft; + attribute stale : Boolean = drift != 0.0; +} +``` + +Each run is then one usage at the top level of the `Records` package — the +same package `-record-run` writes its records into — filled in from the +printed output. The definitions themselves sit in `Records::Vocab`, one +nesting level down: the report's queries walk the package's direct members, +and a `part def` matching `WhereType` would surface its own unbound features +as a bogus row. + +```sysml +part scoutRun : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:14:00Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget"; + kind = "run"; + } + attribute :>> caseName = "Descent::scoutBudget"; + attribute :>> kind = "run"; + attribute :>> 'objective' = "satisfied"; + ref :>> 'subject' = Landers::scout; + ref part :>> lander = scout; + attribute :>> subjectName = "Landers::scout"; + attribute :>> burnTime = 40.0; + attribute :>> fuelUsed = 120.0; + attribute :>> wetMass = 730.0; + attribute :>> fuelLeft = 130.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } +} +``` + +Two limitations shape the record. The annotation's attribute values are +not projectable — `Project(properties = ("runAt"))` reports +`unknown property` — so `command` is also carried as a plain attribute on +`DemoRun` for the provenance table to show; the `@AnalysisRecords::RecordedRun` +metadata still answers `WhereMetadata` filters and keeps the provenance +machine-readable. And the library's untyped `'subject'` ref cannot drive the +`liveFuelLeft` formula, so `FuelBudgetRun` keeps a typed `lander` ref and each +record binds both to the same part. Sweep records set the library's +`iteration` attribute (1, 2, 3); the trade-study record carries its scores as +`EvaluationRecord` usages under `evaluations`, the same shape `-record-run` +emits. + +## The runs that were recorded + +One baseline run per candidate, at `FuelBudget`'s default 40 s burn. **Two of +these transcripts are records, not what the commands print today**: the relay +run and the trade study were captured before `relay.fuel` was edited from +`180.0` to `210.0`, which is what makes the staleness section below possible. +Each is labelled; the scout and hauler runs print the same numbers now. + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget +``` + +``` +✓ Descent::scoutBudget + fuelUsed = 120.0 + wetMass = 730.0 + fuelLeft = 130.0 + objective reserveHeld: satisfied +``` + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::haulerBudget +``` + +``` + fuelUsed = 320.0 + wetMass = 1980.0 + fuelLeft = 580.0 +``` + +The relay run, **as recorded** (when `relay.fuel` was `180.0`): + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::relayBudget +``` + +``` + fuelUsed = 100.0 + wetMass = 530.0 + fuelLeft = 80.0 +``` + +The same command **now prints** — the difference the `relayRun` record +detects: + +``` + fuelUsed = 100.0 + wetMass = 560.0 + fuelLeft = 110.0 +``` + +A sweep of `scoutBudget`, one record per row (this README's tables omit the +`time` column, which is wall time): + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -analysis Descent::scoutBudget -sweep "burnTime=40.0..80.0:20.0" +``` + +``` +burnTime | fuelUsed | wetMass | fuelLeft | verdict +---------+----------+---------+----------+---------------------------- +40.0 | 120.0 | 730.0 | 130.0 | reserveHeld: satisfied +60.0 | 180.0 | 670.0 | 70.0 | reserveHeld: satisfied +80.0 | 240.0 | 610.0 | 10.0 | reserveHeld: not satisfied +``` + +The 80 s row leaves the objective unsatisfied, so the sweep exits `1`; the +transcripts here and above show the result lines only, not the package +loading and `standing` lines every run also prints. + +And the trade study, **as recorded** before the relay edit: + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Selection::lightest +``` + +``` +✓ Selection::lightest + selectedAlternative = Landers::relay (object #3) + objective tradeStudyObjective: satisfied + evaluationFunction(Landers::scout (object #1)) = 850.0 + evaluationFunction(Landers::hauler (object #2)) = 2300.0 + evaluationFunction(Landers::relay (object #3)) = 630.0 [selected] +``` + +The same command **now prints** `660.0` for relay (`450.0 + 210.0`) — the +selection is still `relay`, but the score the record saved no longer is. + +## Records go stale — and can say so + +The `relay` part was edited *after* its run was recorded: `fuel` went from +`180.0` to `210.0` (the attribute carries a comment saying so). The record +still holds the printed `fuelLeft = 80.0`, but `liveFuelLeft` recomputes the +same formula from the model as it now stands — `210.0 - 40.0 * 2.5 = 110.0` — +so `drift = 30.0` and `stale = true`. That is the one thing a printed report +can never give you: a record that notices the model moved. + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -run-query "Reporting::StaleRuns" +``` + +``` +✓ Query Reporting::StaleRuns returned 1 row + Columns: name, subjectName, burnTime, fuelLeft, liveFuelLeft, drift + Row 1: Records::relayRun + fuelLeft = 80.0 + liveFuelLeft = 110.0 + drift = 30.0 +``` + +The relay edit made a second record stale too: `lightestRun`'s third +`EvaluationRecord` still says `score = 630.0` while `-analysis +Selection::lightest` now scores relay `660.0`. Nothing flags it — +`EvaluationRecord` rederives nothing, so it has no `liveFuelLeft` to compare +against. That is the limitation the paragraph above describes, made concrete: +drift detection only exists where the record definition recomputes the value +itself, which neither the library's records nor `-record-run`'s output does — +a recompute-def like `FuelBudgetRun` is something a modeler writes on purpose. + +The comparison is a derived Boolean on the record definition rather than a +`Column` expression, because computed columns do not support `!=`. Any model +edit can produce drift this way — only records typed by a definition that +recomputes an output can detect it, and only for the values it rederives. + +## Rendering the records + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -render-document Reporting::AnalysisReport -o examples/analysis-results-demo/report.md +``` + +[`report.md`](report.md) is committed so the test suite can compare the +render byte-for-byte. It opens with a table of **every recorded run** — +`WhereMetadata(... 'metadata' = "AnalysisRecords::RecordedRun")` filtered to +`WhereType(... type = "AnalysisRecords::AnalysisRun")` over the `Records` +package's direct members — then a grouped table of every fuel-budget record +by subject, the sweep rows alone, the stale-records table (exactly +`relayRun`), a provenance table over the same `WhereMetadata` filter, the +trade-study record and its `EvaluationRecord`s, and — the contrast — a +`Verdicts` table of the assertions about `scout` **evaluated live at render +time**: the records say what a run printed; the verdicts say what holds now. + +HTML and PDF render the same document tree: + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -render-document Reporting::AnalysisReport -doc-form html -o report.html +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -render-document Reporting::AnalysisReport -doc-form pdf -o report.pdf +``` + +## Recording runs automatically + +Everything above was written by hand; the same vocabulary is what +`-record-run` emits. Running a case with `-record-run` records it into a +`Records` package beside the case and, with `-convert sysml`, writes the +model — records included — back out: + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -record-run "Descent::scoutBudget" -convert sysml -o recorded.sysml +``` + +``` +✓ Descent::scoutBudget + fuelUsed = 120.0 + wetMass = 730.0 + fuelLeft = 130.0 + objective reserveHeld: satisfied + standing: value (observed: 1 run under reverse) + recorded Records::scoutBudget_run1 (Records::ScoutBudgetRun) +``` + +The generated record is the same shape this demo writes by hand — a def per +case specializing `AnalysisRecords::AnalysisRun`, the library annotation, the +subject ref, and a `VerdictRecord` per check: + +```sysml +part def ScoutBudgetRun :> AnalysisRecords::AnalysisRun { + attribute :>> caseName default = "Descent::scoutBudget"; + attribute burnTime : ScalarValues::Real; + ... +} +part scoutBudget_run1 : ScoutBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2026-09-24T06:08:21Z"; + tool = "sysml v0.8.1-2129-ge8b389eea"; + command = "-record-run \"Descent::scoutBudget\""; + kind = "run"; + } + ... + ref :>> 'subject' = Landers::scout; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } +} +``` + +`-record-run` targets the `Records` package because it is the plain +top-level package beside `Descent` — the same package this demo's +hand-written records live in, so the generated record lands where the +report's queries already walk. Render the document in the same invocation +and the new record joins it: + +```bash +./bin/sysml examples/analysis-results-demo/lander-results.sysml \ + -record-run "Descent::scoutBudget" \ + -render-document Reporting::AnalysisReport -o recorded-report.md +``` + +The *Every recorded run* table — annotated `AnalysisRun`s, whatever their +definition — picks it up as an eighth row: + +``` +| name | caseName | kind | subjectName | objective | +| --- | --- | --- | --- | --- | +| scoutRun | Descent::scoutBudget | run | Landers::scout | satisfied | +| haulerRun | Descent::haulerBudget | run | Landers::hauler | satisfied | +| relayRun | Descent::relayBudget | run | Landers::relay | satisfied | +| scoutSweep40 | Descent::scoutBudget | sweep | Landers::scout | satisfied | +| scoutSweep60 | Descent::scoutBudget | sweep | Landers::scout | satisfied | +| scoutSweep80 | Descent::scoutBudget | sweep | Landers::scout | not satisfied | +| lightestRun | Selection::lightest | trade | | satisfied | +| scoutBudget\_run1 | Descent::scoutBudget | run | Landers::scout | satisfied | +``` + +The fuel-budget, sweep and stale tables do *not* pick it up — they filter +`WhereType(... type = "FuelBudgetRun")`, and the generated def specializes +`AnalysisRun`, not `FuelBudgetRun` — and the provenance table lists it with +an empty `command` cell, since only the hand-written `DemoRun` declares that +projectable attribute. The REPL form is `%record`; sweeps record one record +per row (`recorded 3 runs as Records::scoutBudget_run1 …`), though `-convert` +refuses a sweep — write the model out after a single `-record-run`, or +record each row explicitly. The full flag reference is +[the manual](../../docs/manual/recording-analysis-runs.md). + +These records stay hand-written for two reasons: the stale-relay story needs +values from *before* `relay.fuel` changed — a `-record-run` today would +record `fuelLeft = 110.0` and no drift — and the derived `liveFuelLeft` / +`drift` / `stale` columns live on the shared `DemoRun`/`FuelBudgetRun` defs, +while `-record-run` writes one def per case with no recompute. + +## Where to read more + +- [analysis-demo](../analysis-demo/README.md): the same lander asked with + `-analysis`, `-sweep`, trade studies and scheduling policies — everything + this demo records. +- [verdicts-demo](../verdicts-demo/README.md): `Verdicts(...)` rows in depth. +- The [query cookbook](../../docs/manual/query-cookbook.md): + [metadata filters](../../docs/manual/query-cookbook.md#metadata-filters), + [property filters](../../docs/manual/query-cookbook.md#property-filters), + [sorting](../../docs/manual/query-cookbook.md#sorting) and + [derived values](../../docs/manual/query-cookbook.md#derived-values), and + the [authoring manual](../../docs/manual/authoring.md) for + `Table`, `groupBy` and `Definitions`. diff --git a/examples/analysis-results-demo/lander-results.sysml b/examples/analysis-results-demo/lander-results.sysml new file mode 100644 index 0000000000..74c4c01054 --- /dev/null +++ b/examples/analysis-results-demo/lander-results.sysml @@ -0,0 +1,547 @@ +// An analysis run's printed results are discarded unless recorded — `%record` +// and `-record-run` write them back on the `AnalysisRecords` library +// vocabulary, or, as here, by hand. These records are hand-written because the +// story needs values from before a model edit: a derived attribute on the +// record recomputes the run's output from the model *as it stands now*, so a +// record whose numbers the model has moved away from reports its own drift. +// See README.md for the commands that produced each recorded row. +package Landers { + private import ScalarValues::*; + + part def Lander { + attribute dryMass : Real; // kg, without fuel + attribute fuel : Real; // kg loaded at the start of descent + attribute burnRate : Real; // kg/s the engine consumes at full thrust + attribute payload : Real; // kg of instruments carried + attribute touchdownSpeed : Real; // m/s, from the descent profile + } + + part scout : Lander { + attribute :>> dryMass = 600.0; + attribute :>> fuel = 250.0; + attribute :>> burnRate = 3.0; + attribute :>> payload = 40.0; + attribute :>> touchdownSpeed = 1.2; + } + part hauler : Lander { + attribute :>> dryMass = 1400.0; + attribute :>> fuel = 900.0; + attribute :>> burnRate = 8.0; + attribute :>> payload = 300.0; + attribute :>> touchdownSpeed = 1.9; + } + part relay : Lander { + attribute :>> dryMass = 450.0; + // The recorded relay run and trade study predate this edit: fuel was + // 180.0 then, so relayRun reads stale and lightestRun silently disagrees. + attribute :>> fuel = 210.0; + attribute :>> burnRate = 2.5; + attribute :>> payload = 25.0; + attribute :>> touchdownSpeed = 1.4; + } +} + +package Descent { + private import ScalarValues::*; + private import Landers::*; + + // The requirement the analysis objective is typed by; the case binds its subject. + requirement def FuelReserve { + subject lander : Lander; + in attribute fuelLeft : Real; + attribute reserve : Real = 20.0; + require constraint { fuelLeft >= reserve } + } + + // Fuel a descent burn leaves: `then`-chained action steps feed each other + // through parameter bindings, and the case's outputs are read afterwards. + analysis def FuelBudget { + subject lander : Lander; + in attribute burnTime : Real = 40.0; // s of engine burn + action burn { + in rate : Real = lander.burnRate; + in seconds : Real = burnTime; + out used : Real; + assign used := rate * seconds; + } + then action remaining { + in loaded : Real = lander.fuel; + in used : Real = burn.used; + out left : Real; + assign left := loaded - used; + } + + out fuelUsed : Real = burn.used; + out wetMass : Real = lander.dryMass + remaining.left; + + objective reserveHeld : FuelReserve { + subject = lander; + in fuelLeft = remaining.left; + } + + return fuelLeft : Real = remaining.left; + } + + analysis scoutBudget : FuelBudget { + subject lander = scout; + } + analysis haulerBudget : FuelBudget { + subject lander = hauler; + } + analysis relayBudget : FuelBudget { + subject lander = relay; + } + + // The touchdown requirement, stated of each candidate and satisfied by it. + requirement def SoftLanding { + subject lander : Lander; + attribute limit : Real = 1.5; + require constraint { lander.touchdownSpeed <= limit } + } + requirement scoutLandsSoftly : SoftLanding; + satisfy scoutLandsSoftly by scout; + requirement haulerLandsSoftly : SoftLanding; + satisfy haulerLandsSoftly by hauler; + requirement relayLandsSoftly : SoftLanding; + satisfy relayLandsSoftly by relay; + + // Verifies the requirement from its objective; `PassIf` decides the verdict. + verification def CheckScout { + subject lander : Lander; + objective { + verify scoutLandsSoftly; + require constraint { lander.touchdownSpeed <= 1.5 } + } + VerificationCases::PassIf(lander.touchdownSpeed <= 1.5) + } + verification def CheckHauler { + subject lander : Lander; + objective { + verify haulerLandsSoftly; + require constraint { lander.touchdownSpeed <= 1.5 } + } + VerificationCases::PassIf(lander.touchdownSpeed <= 1.5) + } + verification def CheckRelay { + subject lander : Lander; + objective { + verify relayLandsSoftly; + require constraint { lander.touchdownSpeed <= 1.5 } + } + VerificationCases::PassIf(lander.touchdownSpeed <= 1.5) + } + verification checkScout : CheckScout { + subject lander = scout; + } + verification checkHauler : CheckHauler { + subject lander = hauler; + } + verification checkRelay : CheckRelay { + subject lander = relay; + } +} + +package Selection { + private import ScalarValues::*; + private import TradeStudies::*; + private import Landers::*; + + // Lightest on the pad: every alternative is scored, and the objective picks the least. + analysis lightest : TradeStudy { + subject : Lander[1..*] = (scout, hauler, relay); + objective : MinimizeObjective; + calc :>> evaluationFunction { + in part l :>> alternative : Lander; + return :>> result : Real = l.dryMass + l.fuel; + } + return part :>> selectedAlternative : Lander; + } +} + +package Records { + private import ScalarValues::*; + private import Landers::*; + private import AnalysisRecords::*; + + // The record definitions live one nesting level down: queries walk the + // package's direct members, and a `part def` matching `WhereType` would + // surface its own unbound features. + private package Vocab { + // Shared definition for the demo's hand-written records: the library's + // AnalysisRun plus `command` as a plain attribute — a document query's + // `Project` cannot read an annotation's attribute values, so the + // provenance table needs it declared. + part def DemoRun :> AnalysisRecords::AnalysisRun { + attribute command : String; + } + + // A FuelBudget run written back: subject, input and outputs as + // recorded, plus the output recomputed from the model as it stands + // now. `drift` measures how far the model has moved since the record + // was written, and `stale` names a record that no longer matches. + // `lander` is the typed subject the formula reads; the library's + // untyped `'subject'` ref is bound to the same part on each record. + part def FuelBudgetRun :> DemoRun { + ref part lander : Lander; + attribute burnTime : Real; + attribute fuelUsed : Real; + attribute wetMass : Real; + attribute fuelLeft : Real; + attribute liveFuelLeft : Real = lander.fuel - burnTime * lander.burnRate; + attribute drift : Real = liveFuelLeft - fuelLeft; + attribute stale : Boolean = drift != 0.0; + } + + // A TradeStudy run written back: the selection, plus the run's checks + // and evaluations carried under the library's `verdicts` and + // `evaluations`. + part def TradeStudyRun :> DemoRun { + attribute selected : String; + } + } + private import Vocab::*; + // One baseline run per candidate, at FuelBudget's default 40 s burn. + part scoutRun : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:14:00Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget"; + kind = "run"; + } + attribute :>> caseName = "Descent::scoutBudget"; + attribute :>> kind = "run"; + attribute :>> 'objective' = "satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget"; + ref :>> 'subject' = Landers::scout; + ref part :>> lander = scout; + attribute :>> subjectName = "Landers::scout"; + attribute :>> burnTime = 40.0; + attribute :>> fuelUsed = 120.0; + attribute :>> wetMass = 730.0; + attribute :>> fuelLeft = 130.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } + } + part haulerRun : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:14:30Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::haulerBudget"; + kind = "run"; + } + attribute :>> caseName = "Descent::haulerBudget"; + attribute :>> kind = "run"; + attribute :>> 'objective' = "satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::haulerBudget"; + ref :>> 'subject' = Landers::hauler; + ref part :>> lander = hauler; + attribute :>> subjectName = "Landers::hauler"; + attribute :>> burnTime = 40.0; + attribute :>> fuelUsed = 320.0; + attribute :>> wetMass = 1980.0; + attribute :>> fuelLeft = 580.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } + } + part relayRun : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:15:00Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::relayBudget"; + kind = "run"; + } + attribute :>> caseName = "Descent::relayBudget"; + attribute :>> kind = "run"; + attribute :>> 'objective' = "satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::relayBudget"; + ref :>> 'subject' = Landers::relay; + ref part :>> lander = relay; + attribute :>> subjectName = "Landers::relay"; + attribute :>> burnTime = 40.0; + attribute :>> fuelUsed = 100.0; + attribute :>> wetMass = 530.0; + attribute :>> fuelLeft = 80.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } + } + + // A sweep of scoutBudget over burnTime, one record per row the sweep printed. + part scoutSweep40 : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:20:10Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0"; + kind = "sweep"; + } + attribute :>> caseName = "Descent::scoutBudget"; + attribute :>> kind = "sweep"; + attribute :>> 'objective' = "satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0"; + ref :>> 'subject' = Landers::scout; + ref part :>> lander = scout; + attribute :>> subjectName = "Landers::scout"; + attribute :>> iteration = 1; + attribute :>> burnTime = 40.0; + attribute :>> fuelUsed = 120.0; + attribute :>> wetMass = 730.0; + attribute :>> fuelLeft = 130.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } + } + part scoutSweep60 : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:20:10Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0"; + kind = "sweep"; + } + attribute :>> caseName = "Descent::scoutBudget"; + attribute :>> kind = "sweep"; + attribute :>> 'objective' = "satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0"; + ref :>> 'subject' = Landers::scout; + ref part :>> lander = scout; + attribute :>> subjectName = "Landers::scout"; + attribute :>> iteration = 2; + attribute :>> burnTime = 60.0; + attribute :>> fuelUsed = 180.0; + attribute :>> wetMass = 670.0; + attribute :>> fuelLeft = 70.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "satisfied"; + } + } + part scoutSweep80 : FuelBudgetRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:20:10Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0"; + kind = "sweep"; + } + attribute :>> caseName = "Descent::scoutBudget"; + attribute :>> kind = "sweep"; + attribute :>> 'objective' = "not satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0"; + ref :>> 'subject' = Landers::scout; + ref part :>> lander = scout; + attribute :>> subjectName = "Landers::scout"; + attribute :>> iteration = 3; + attribute :>> burnTime = 80.0; + attribute :>> fuelUsed = 240.0; + attribute :>> wetMass = 610.0; + attribute :>> fuelLeft = 10.0; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "reserveHeld"; + attribute :>> status = "not satisfied"; + } + } + + // The trade study run: the selection, its objective verdict, and each + // alternative's evaluation — recorded before relay.fuel changed, so the + // relay score here is the old 630.0. + part lightestRun : TradeStudyRun { + @AnalysisRecords::RecordedRun { + runAt = "2025-11-02T09:25:00Z"; + tool = "sysml"; + command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Selection::lightest"; + kind = "trade"; + } + attribute :>> caseName = "Selection::lightest"; + attribute :>> kind = "trade"; + attribute :>> 'objective' = "satisfied"; + attribute :>> command = "./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Selection::lightest"; + attribute :>> selected = "Landers::relay"; + part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { + attribute :>> kind = "objective"; + attribute :>> name = "tradeStudyObjective"; + attribute :>> status = "satisfied"; + } + part evaluation1 : AnalysisRecords::EvaluationRecord :> evaluations { + attribute :>> 'function' = "Selection::lightest::evaluationFunction"; + attribute :>> alternative = "Landers::scout (object #1)"; + attribute :>> score = 850.0; + attribute :>> result = "850.0"; + attribute :>> selected = false; + attribute :>> tied = false; + } + part evaluation2 : AnalysisRecords::EvaluationRecord :> evaluations { + attribute :>> 'function' = "Selection::lightest::evaluationFunction"; + attribute :>> alternative = "Landers::hauler (object #2)"; + attribute :>> score = 2300.0; + attribute :>> result = "2300.0"; + attribute :>> selected = false; + attribute :>> tied = false; + } + part evaluation3 : AnalysisRecords::EvaluationRecord :> evaluations { + attribute :>> 'function' = "Selection::lightest::evaluationFunction"; + attribute :>> alternative = "Landers::relay (object #3)"; + attribute :>> score = 630.0; + attribute :>> result = "630.0"; + attribute :>> selected = true; + attribute :>> tied = false; + } + } +} + +package Reporting { + private import ScalarValues::*; + private import DocumentQueries::*; + private import KerML::Root::Element; + + // The records package's members walked by name, so a run `-record-run` + // writes into the same `Records` package joins the query results. + calc def AllRuns :> Query { + Project( + source = WhereType( + source = WhereMetadata( + source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), + 'metadata' = "AnalysisRecords::RecordedRun"), + type = "AnalysisRecords::AnalysisRun"), + properties = ("name", "caseName", "kind", "subjectName", "objective")) + } + + // Every saved fuel-budget run, sorted by burn time; the document groups it by subject. + calc def FuelBudgetRuns :> Query { + Project( + source = OrderBy( + source = WhereType(source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), type = "FuelBudgetRun"), + property = "burnTime", direction = "ascending", missing = "last", multiple = "error"), + properties = ("name", "subjectName", "kind", "burnTime", "fuelUsed", "wetMass", "fuelLeft", "objective")) + } + + // The sweep rows alone. + calc def SweepRuns :> Query { + Project( + source = OrderBy( + source = WhereFeature( + source = WhereType(source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), type = "FuelBudgetRun"), + 'feature' = "kind", operator = "=", value = "sweep"), + property = "burnTime", direction = "ascending", missing = "last", multiple = "error"), + properties = ("name", "caseName", "burnTime", "fuelUsed", "fuelLeft", "objective")) + } + + // Records whose saved numbers no longer match the model as it stands. + calc def StaleRuns :> Query { + Project( + source = WhereFeature( + source = WhereType(source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), type = "FuelBudgetRun"), + 'feature' = "stale", operator = "=", value = "true"), + properties = ("name", "subjectName", "burnTime", "fuelLeft", "liveFuelLeft", "drift")) + } + + // The recorded runs with their provenance annotation's command line. + calc def Provenance :> Query { + Project( + source = WhereMetadata(source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), 'metadata' = "AnalysisRecords::RecordedRun"), + properties = ("name", "kind", "caseName", "command")) + } + + // The trade-study record. + calc def TradeRuns :> Query { + Project( + source = WhereType(source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), type = "TradeStudyRun"), + properties = ("name", "caseName", "selected", "objective")) + } + + // The evaluations nested under the trade-study record. + calc def TradeEvaluations :> Query { + Project( + source = WhereType(source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 2), type = "EvaluationRecord"), + properties = ("alternative", "score", "selected", "tied")) + } + + // Assertions about the scout, checked live at render time. + calc def ScoutVerdicts :> Query { + in root : Element; + Project( + source = Verdicts(source = root), + properties = ("kind", "name", "verdict", "reason")) + } + + // The document itself. + part def AnalysisReport :> Document { + attribute redefines title = "Recorded analysis runs"; + + part intro : Paragraph { + attribute redefines text = "An analysis run's printed results are discarded unless they are recorded — by -record-run or by hand, in the vocabulary the AnalysisRecords library defines. Every row below is a declared record typed by a run definition on that vocabulary, annotated with the command that produced it — except the last table, whose verdicts are recomputed live."; + } + + part every : Section { + attribute redefines title = "Every recorded run"; + part runs : Table { + attribute redefines caption = "Every @AnalysisRecords::RecordedRun-annotated AnalysisRun in the Records package"; + calc rows : AllRuns; + } + } + + part budgets : Section { + attribute redefines title = "Recorded fuel budgets"; + part runs : Table { + attribute redefines caption = "One record per run, grouped by subject"; + attribute redefines groupBy = "subjectName"; + calc rows : FuelBudgetRuns; + } + } + + part sweep : Section { + attribute redefines title = "Sweep of scoutBudget"; + part runs : Table { + attribute redefines caption = "One record per sweep row"; + calc rows : SweepRuns; + } + } + + part stale : Section { + attribute redefines title = "Runs that no longer match the model"; + part runs : Table { + attribute redefines caption = "Records whose saved fuelLeft differs from the value the model now derives"; + calc rows : StaleRuns; + } + } + + part provenance : Section { + attribute redefines title = "Provenance"; + part runs : Table { + attribute redefines caption = "Every element annotated @AnalysisRecords::RecordedRun"; + calc rows : Provenance; + } + } + + part trade : Section { + attribute redefines title = "Trade study"; + part runs : Table { + attribute redefines caption = "The recorded selection and verdict"; + calc rows : TradeRuns; + } + part evals : Table { + attribute redefines caption = "Each alternative's evaluation, as recorded"; + calc rows : TradeEvaluations; + } + } + + part live : Section { + attribute redefines title = "Live verdicts"; + part runs : Table { + attribute redefines caption = "Assertions about the scout, evaluated at render time"; + calc rows : ScoutVerdicts { + in root = Landers::scout; + } + } + } + } +} diff --git a/examples/analysis-results-demo/report.md b/examples/analysis-results-demo/report.md new file mode 100644 index 0000000000..f8b8209f46 --- /dev/null +++ b/examples/analysis-results-demo/report.md @@ -0,0 +1,99 @@ +# Recorded analysis runs + +An analysis run's printed results are discarded unless they are recorded — by -record-run or by hand, in the vocabulary the AnalysisRecords library defines. Every row below is a declared record typed by a run definition on that vocabulary, annotated with the command that produced it — except the last table, whose verdicts are recomputed live. + +## Every recorded run + +*Every @AnalysisRecords::RecordedRun-annotated AnalysisRun in the Records package* + +| name | caseName | kind | subjectName | objective | +| --- | --- | --- | --- | --- | +| scoutRun | Descent::scoutBudget | run | Landers::scout | satisfied | +| haulerRun | Descent::haulerBudget | run | Landers::hauler | satisfied | +| relayRun | Descent::relayBudget | run | Landers::relay | satisfied | +| scoutSweep40 | Descent::scoutBudget | sweep | Landers::scout | satisfied | +| scoutSweep60 | Descent::scoutBudget | sweep | Landers::scout | satisfied | +| scoutSweep80 | Descent::scoutBudget | sweep | Landers::scout | not satisfied | +| lightestRun | Selection::lightest | trade | | satisfied | + +## Recorded fuel budgets + +*One record per run, grouped by subject* + +**subjectName: Landers::scout** + +| name | subjectName | kind | burnTime | fuelUsed | wetMass | fuelLeft | objective | +| --- | --- | --- | --- | --- | --- | --- | --- | +| scoutRun | Landers::scout | run | 40 | 120 | 730 | 130 | satisfied | +| scoutSweep40 | Landers::scout | sweep | 40 | 120 | 730 | 130 | satisfied | +| scoutSweep60 | Landers::scout | sweep | 60 | 180 | 670 | 70 | satisfied | +| scoutSweep80 | Landers::scout | sweep | 80 | 240 | 610 | 10 | not satisfied | + +**subjectName: Landers::hauler** + +| name | subjectName | kind | burnTime | fuelUsed | wetMass | fuelLeft | objective | +| --- | --- | --- | --- | --- | --- | --- | --- | +| haulerRun | Landers::hauler | run | 40 | 320 | 1980 | 580 | satisfied | + +**subjectName: Landers::relay** + +| name | subjectName | kind | burnTime | fuelUsed | wetMass | fuelLeft | objective | +| --- | --- | --- | --- | --- | --- | --- | --- | +| relayRun | Landers::relay | run | 40 | 100 | 530 | 80 | satisfied | + +## Sweep of scoutBudget + +*One record per sweep row* + +| name | caseName | burnTime | fuelUsed | fuelLeft | objective | +| --- | --- | --- | --- | --- | --- | +| scoutSweep40 | Descent::scoutBudget | 40 | 120 | 130 | satisfied | +| scoutSweep60 | Descent::scoutBudget | 60 | 180 | 70 | satisfied | +| scoutSweep80 | Descent::scoutBudget | 80 | 240 | 10 | not satisfied | + +## Runs that no longer match the model + +*Records whose saved fuelLeft differs from the value the model now derives* + +| name | subjectName | burnTime | fuelLeft | liveFuelLeft | drift | +| --- | --- | --- | --- | --- | --- | +| relayRun | Landers::relay | 40 | 80 | 110 | 30 | + +## Provenance + +*Every element annotated @AnalysisRecords::RecordedRun* + +| name | kind | caseName | command | +| --- | --- | --- | --- | +| scoutRun | run | Descent::scoutBudget | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget | +| haulerRun | run | Descent::haulerBudget | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::haulerBudget | +| relayRun | run | Descent::relayBudget | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::relayBudget | +| scoutSweep40 | sweep | Descent::scoutBudget | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0 | +| scoutSweep60 | sweep | Descent::scoutBudget | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0 | +| scoutSweep80 | sweep | Descent::scoutBudget | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Descent::scoutBudget -sweep burnTime=40.0..80.0:20.0 | +| lightestRun | trade | Selection::lightest | ./bin/sysml examples/analysis-results-demo/lander-results.sysml -analysis Selection::lightest | + +## Trade study + +*The recorded selection and verdict* + +| name | caseName | selected | objective | +| --- | --- | --- | --- | +| lightestRun | Selection::lightest | Landers::relay | satisfied | + +*Each alternative's evaluation, as recorded* + +| alternative | score | selected | tied | +| --- | --- | --- | --- | +| Landers::scout (object \#1) | 850 | false | false | +| Landers::hauler (object \#2) | 2300 | false | false | +| Landers::relay (object \#3) | 630 | true | false | + +## Live verdicts + +*Assertions about the scout, evaluated at render time* + +| kind | name | verdict | reason | +| --- | --- | --- | --- | +| satisfaction | | holds | | +| verification | checkScout | holds | | diff --git a/tests/corpus/testdata/api_json_roundtrip_expected.txt b/tests/corpus/testdata/api_json_roundtrip_expected.txt index 5d2ee4f305..ba2a430968 100644 --- a/tests/corpus/testdata/api_json_roundtrip_expected.txt +++ b/tests/corpus/testdata/api_json_roundtrip_expected.txt @@ -9,13 +9,14 @@ # This is a per-file ratchet, not a claim that any verdict is right; # see docs/project/rdf-corpus-roundtrip.md. Regenerate with: # go test ./tests/corpus -run TestCorpusAPIJSONRoundTrip -update-api-json-roundtrip -# files: committed 43 +# files: committed 44 # files: sysml-v2-training 100 # files: pilot-corpora/kerml-examples 58 # files: pilot-corpora/sysml-examples 99 # files: pilot-corpora/sysml-validation 56 stable action-executor-demo.sysml stable analysis-demo/lander.sysml +stable analysis-results-demo/lander-results.sysml stable combined-behavioral-demo.sysml stable disposal-robot-demo/robot.sysml stable disposal-team-demo/team.sysml diff --git a/tests/corpus/testdata/corpus_roundtrip_expected.txt b/tests/corpus/testdata/corpus_roundtrip_expected.txt index e173edf202..12a9b13540 100644 --- a/tests/corpus/testdata/corpus_roundtrip_expected.txt +++ b/tests/corpus/testdata/corpus_roundtrip_expected.txt @@ -8,13 +8,14 @@ # is a per-file ratchet, not a claim that any verdict is right; see # docs/project/rdf-corpus-roundtrip.md. Regenerate with: # go test ./tests/corpus -run TestCorpusRoundTrip -update-corpus-roundtrip -# files: committed 43 +# files: committed 44 # files: sysml-v2-training 100 # files: pilot-corpora/kerml-examples 58 # files: pilot-corpora/sysml-examples 99 # files: pilot-corpora/sysml-validation 56 stable action-executor-demo.sysml stable analysis-demo/lander.sysml +stable analysis-results-demo/lander-results.sysml stable combined-behavioral-demo.sysml stable disposal-robot-demo/robot.sysml stable disposal-team-demo/team.sysml