docs(examples): record analysis runs in the model and report them in a document - #556
Merged
Merged
Conversation
…a document Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
…demo Co-Authored-By: jason.han <hanhuijun@gmail.com>
… example Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Contributor
Author
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
🔍 Devin Review: 2 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…s library Co-Authored-By: jason.han <hanhuijun@gmail.com>
…sed demo Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ge so recorded runs join it Co-Authored-By: jason.han <hanhuijun@gmail.com>
…oted demo Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Nothing in the tree showed how a run's results get into a generated document. This adds
examples/analysis-results-demo/, a worked example in which analysis runs are recorded into the model aspartusages on the bundledAnalysisRecordsvocabulary — the same shape%record/-record-runemits — and a document tabulates the records.The top-level
Recordspackage —-record-run's default target besideDescent— holds seven records (three baseline runs, a three-point sweep withiteration, one trade study whose scores areEvaluationRecordparts underevaluations) whose numbers are the tool's actual output;Reporting::AnalysisReportwalksDescendants(Named("Records")), opens with an "Every recorded run" table filtered onAnalysisRecords::AnalysisRun, groups the fuel budgets by subject, isolates the sweep rows, lists provenance overWhereMetadata("AnalysisRecords::RecordedRun"), shows the trade study and its evaluations, and contrasts them withVerdictsrecomputed at render time.relay.fuelwas deliberately edited after recording, so exactlyrelayRunreportsstale = true, drift = 30.0, and the README shows why the trade record silently disagrees instead (it rederives nothing).The records are hand-written rather than produced by
-record-runbecause the stale-relay story needs values from before the model edit and the deriveddrift/stalecolumns live on a shared def, while-record-runwrites one def per case with no recompute. The README shows the-record-runcommand over this model and the record it generates, and the report rendered with-record-run "Descent::scoutBudget"shows the generatedscoutBudget_run1beside the hand-written rows in the all-runs and Provenance tables (theFuelBudgetRun-filtered tables exclude it, as documented).Limitations the example documents: metadata annotation attributes are not projectable (
commandis duplicated as a plain attribute onDemoRun), the library's'subject'ref is untyped so the drift formula reads a typedlanderref,objectiveis a reserved word, computedColumnexpressions do not support!=, and apart defmatching aWhereTypefilter surfaces its own unbound features (the defs live in a nestedVocabpackage).Also: a Walkthroughs row in
examples/README.md, a paragraph indocs/manual/query-kinds.mdpointing at the demo, a changelog fragment, and the re-recorded pilot-differential baseline and generated figures (the new file adds 397 pilot-only diagnostics, all on document-query calls andAnalysisRecordsconstructs the pinned pilot has no support for; adjudicated indocs/project/pilot-differential.md).How it was verified
TestAnalysisResultsExample(cmd/sysml): the render matches the committedreport.mdbyte-for-byte, a render with-record-run "Descent::scoutBudget"listsscoutBudget_run1in the all-runs and Provenance tables with the stale table unchanged,Reporting::StaleRunsreturns exactlyrelayRunwithdrift = 30.0, and-analysis Descent::scoutBudgetstill prints thefuelLeftthe scout record saved.TestCorpusRoundTrip/ API-JSON round trip: the new model isstable; the only expectation change is its added line.go run -C tools ./cmd/pilot-diff -update,doc-counts, andgo test ./referee/diff ./referee/xpect ./census/doccountsintools.go build ./... && go vet ./...,gofmt -l .empty,./bin/sysml … -validateclean,scripts/check-doc-links.py,scripts/check-doc-ids.py,scripts/changelog.py check.Checklist
make testandmake lintpass locallychanges/unreleased/<slug>.<section>.md, not as an edit toCHANGELOG.mdmake docs-countsrun if a gate count moved (compliance rows need nothing: the census is counted at docs build)F4,K5) in the body, docs, or changelogLink to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/16d2adef11b1413e9656e2d67500f099
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/16d2adef11b1413e9656e2d67500f099?variant=devin
Requested by: @HuiJun