Skip to content

feat(analysis): record analysis runs into the model with %record and -record-run - #557

Merged
HuiJun merged 48 commits into
developfrom
feature/record-analysis-runs
Sep 24, 2026
Merged

HuiJun merged 48 commits into
developfrom
feature/record-analysis-runs

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What and why

-analysis / %analysis, -sweep and -runs printed their results and discarded them: nothing wrote a run back into the model, %save never contained it, and -render-document refused to run alongside an analysis. This adds a first-class "record this run" step that turns a completed analysis run into ordinary declared model elements, submitted through the normal REPL declaration path, so that %save/-convert sysml -o persist them and document queries table, filter and sort them like any other element.

Surface

  • REPL: %record <case>[(<args>)] [<object>] [into <package>]
  • CLI: -record-run "<call>" (repeatable; takes the -analysis call syntax, composes with -sweep and -runs/-seed) and -record-into <package>. -record-run is allowed together with -render-document (-o, -doc-form) and with -convert sysml -o saved.sysml; the records are made first, so the document sees them and the written file holds them. -render-document with plain -analysis still refuses as before; -record-into without -record-run refuses.

Generated record shape (bundled library AnalysisRecords, non-normative, beside DocumentQueries/Simulation)

package Records {                                   // default: a Records package beside the case's package
    part def ScoutBudgetRun :> AnalysisRecords::AnalysisRun {   // created on first use, reused after
        attribute :>> caseName default = "Descent::scoutBudget";  // owner; a same-named sibling case gets Owner_caseRun
        attribute burnTime : ScalarValues::Real;    // every in parameter, then every out/return
        attribute fuelUsed : ScalarValues::Real;
    }
    part scoutBudget_run1 : ScoutBudgetRun {        // _run2, _run3 … deterministic, probed against the model
        @AnalysisRecords::RecordedRun {
            runAt = "2026-01-01T00:00:00Z"; tool = "sysml 0.0.9";
            command = "%record Descent::scoutBudget(3.0) Descent::scout"; kind = "run";
        }
        attribute :>> caseName = "Descent::scoutBudget";
        attribute :>> kind = "run";                 // run | trade | sweep | runs | sample (a -runs sample's conclusion: statistics, result, sample verdicts)
        attribute :>> 'objective' = "satisfied";    // satisfied | not satisfied | undecided
        ref :>> 'subject' = Descent::scout;         // when the subject is a declared usage
        attribute :>> subjectName = "Descent::scout";
        attribute :>> burnTime = 3.0;
        attribute :>> fuelUsed = 12.5;
        part verdict1 : AnalysisRecords::VerdictRecord :> verdicts { … kind, name, status, detail }
        part evaluation1 : AnalysisRecords::EvaluationRecord :> evaluations { … function, alternative, score, result, selected, tied, error }
    }
}

Sweeps and Monte Carlo samples write one record per row/run (iteration set), sharing the provenance. Inputs are the values the body actually ran with — positional, named, defaulted, and the sweep/sample overlay — carried by a new AnalysisResult.Inputs (SweepRow.Inputs, MonteCarloRun.Inputs/Outputs).

Value coverage: Integer/Real/Boolean literals; String (escaped); enumeration literals typed by their enum; quantities as a Real magnitude plus a <name>Unit : String companion; objects as ref to their declared usage; null/unset declared but left unredefined (objective stays as the runtime decided it, undecided when the case declares none); everything else (sequences, arrays, complex, functions, unresolvable objects, Infinity) as a String holding the runtime's printed text — documented as the fallback.

Atomicity: generation is pure (internal/exec/analysis/record.Generate) and validated before anything is submitted. A missing case, a calc, a failed run, a case with no outputs, a per-case definition that does not specialize AnalysisRun or whose attributes have incompatible types, a reserved-name collision, or a submission that would drop declarations or raise new diagnostics all report a diagnostic and leave the model byte-identical; a rollback goes through the same rebuild as a submission, so held objects and debugger sessions survive.

The record merges into a package that a loaded file declared (the one case where the REPL folds into a file snippet), keeping the file's identity so a later reload still supersedes it.

Limitations (documented): a declared name containing :: is ambiguous in the index's raw-joined FQN form (names needing quotes otherwise work); structured values are recorded as their printed text, not nested attributes; the gRPC service is not extended with a record RPC (follow-up); the hand-written examples/analysis-results-demo stays as is and can switch to -record-run in a follow-up — the vocabulary keeps its names (RecordedRun, AnalysisRun, caseName, kind, objective, a ref subject).

How it was verified

  • internal/exec/analysis/record: golden tests (single run with every value kind, trade study, sweep reusing an existing definition) plus error cases; goldens are fixed points of the repository formatter.
  • internal/exec/runtime: AnalysisResult.Inputs order and provenance (positional/named/default), sweep overlay, Monte Carlo inputs/outputs.
  • internal/frontend/repl: %record → %save → reload → %record again yields _run2; document queries over the records (WhereMetadata('metadata' = "AnalysisRecords::RecordedRun"), WhereFeature on objective, Project of outputs); failure leaves the text unchanged and keeps objects and %action debugger sessions; default package placement by the case's owner chain; into a nested package.
  • cmd/sysml: -record-run alone, with -sweep (one row per value), with -runs/-seed, with -render-document (rendered table shows the recorded values), with -convert sysml -o (re-validates clean, re-recording gives _run2), and both refusals.
  • Gate: go build, go vet, gofmt -l empty, make man-check, doc link/ID/changelog checks, mkdocs build --strict, and OPENSYSML_REQUIRE_TRAINING_CORPUS=1 OPENSYSML_REQUIRE_PILOT_CORPORA=1 go test -count=1 ./... green; no corpus ratchet moved. The bundled-library count moved 104 → 105 (self-model figures and architecture doc updated).

Checklist

  • make test and make lint pass locally
  • Tests added or updated for the change
  • Documentation extended where it already covers the surface (see CONTRIBUTING.md)
  • Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved (compliance rows need nothing: the census is counted at docs build)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

devin-ai-integration Bot and others added 9 commits September 24, 2026 01:46
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…lback

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ed library

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration Bot and others added 2 commits September 24, 2026 02:36
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review September 24, 2026 03:12
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 4 commits September 24, 2026 03:25
…n-free

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…nresolved

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ded conversions

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 6 commits September 24, 2026 03:40
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…r gaps

Co-Authored-By: jason.han <hanhuijun@gmail.com>
… was not recorded

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…rries a unit

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits September 24, 2026 03:51
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…nition

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration Bot and others added 4 commits September 24, 2026 04:15
…othing

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 3 commits September 24, 2026 04:26
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…oncrete type

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 5 commits September 24, 2026 04:38
An output read that fails because the activation never assigned it now
carries which output of which calc could not be read, so a Monte Carlo
run can tell an output that depends on a statistic the sample has not
supplied — the sample's own — from one that failed for a reason of its
own.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
The run's record carries the body's verdict as the new AnalysisRun
feature 'verdict', and each verdict the body and its subcases produced
as a VerdictRecord row of kind 'verification' or 'subcase' — they were
run but never recorded. Verifications alone satisfy 'nothing to record'.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
The runtime now leaves the sample's outputs out of a run's Unread, so
the name adjudication against the conclusion is gone — anything left
failed the iteration. A verification case's record passes its verdicts
through. And a generated record adding an occurrence of a diagnostic
the model already reported is refused, not accepted.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 3 commits September 24, 2026 04:51
A name bound on both sides is one parameter, not an input colliding with
an output: the member carries the value the run left in it and a <name>In
companion the value it was bound with, emitted for every inout whatever
the two values. Same-side duplicates and a member colliding with the
companion still error.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
When several loaded files open the same top namespace, the record goes
to the one already holding the deepest prefix of the generated package
nesting, rather than making a second package of that name in the first.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 4 commits September 24, 2026 05:03
… to Real

A scalar-valued enumeration literal records as the literal it is, not the
scalar it equals, and members supplied as Integer and Real alike settle
to Real — Integer literals remaining valid under it, both in a run's own
settlement and against a declared Real member; a declared Integer still
refuses a Real, which would widen a definition the model owns.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ows it

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits September 24, 2026 05:12
…y member

A member supplied as Integer or Real and as a quantity, in either order,
settles to the quantity shape — Real with the unit companion — the plain
row keeping its literal and taking no unit, as a Real-then-quantity row
always did. The inout companion check settles the same pair.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
-schedule, -clock-step, -engine and -instantiate change what the run
computed or which objects it ran on, so the command the record reports
carries each as written, not just the sweep and sample flags.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…anions

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@HuiJun
HuiJun merged commit 5dd5a91 into develop Sep 24, 2026
15 checks passed
@HuiJun
HuiJun deleted the feature/record-analysis-runs branch September 24, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant