scrimlet-reconcilers status (PR 3/3): add omdb command and bootstrap-lockstep API to display status - #11058
Open
jgallagher wants to merge 10 commits into
Open
scrimlet-reconcilers status (PR 3/3): add omdb command and bootstrap-lockstep API to display status#11058jgallagher wants to merge 10 commits into
jgallagher wants to merge 10 commits into
Conversation
Contributor
Author
|
This command requires specifying the bootstrap address of a sled-agent; the easiest thing to do from a switch zone is to ask your own sled-agent, for multiple reasons:
This branch deployed on Asking for the other scrimlet's status is fine too, after finding its IP: Asking for the status from a non-scrimlet sled "works" but there's nothing interesting there: |
jgallagher
force-pushed
the
john/collapse-enum-variants
branch
from
August 12, 2026 15:14
c7d7161 to
e407b6d
Compare
jgallagher
force-pushed
the
john/scrimlet-reconcilers-status
branch
from
August 12, 2026 17:26
af9ea45 to
8b5c9db
Compare
karencfv
reviewed
Aug 12, 2026
| ScrimletReconcilersStatus::WaitingForSledAgentNetworkingInfo => { | ||
| write!( | ||
| f, | ||
| "not running: sled-agent has not yet initialized \ |
Contributor
There was a problem hiding this comment.
Shouldn't this be not yet running if we're still waiting?
Contributor
Author
There was a problem hiding this comment.
Not necessarily; we may never start running (e.g., if we're not a scrimlet).
Contributor
There was a problem hiding this comment.
Then how about changing the sled-agent has not yet initialized bit then? It implies that it will start running eventually no?
Contributor
Author
There was a problem hiding this comment.
Hm, I don't think so, but I can see how you could think that. The actual state transitions we go through here are:
- Waiting for sled-agent to tell us two things (all sleds start in this state, including non-scrimlets):
a. Our underlay networking info (all sleds provide this information, including non-scrimlets)
b. That we have an attached switch and therefore a switch zone (only scrimlets ever set this)
All sleds start out in 1 - that's what this WaitingForSledAgentNetworkingInfo state is. Once a non-scrimlet gets the networking info (1a), it will land in the DeterminingSwitchSlot(NotScrimlet) case, which (correctly, I think) reports itself as not running: no switch detected. The question here is what to print when we're still in state 1: we don't yet know whether we're a scrimlet, so we don't know whether we'll ever start running, but we do know that we'll eventually get 1a - that's where the "sled agent has not yet initialized" verbiage comes in.
All that said, I don't think the exact wording here matters much at all - this is a transient state that will almost never be observed, unless something has gone horribly wrong in which case the exact wording won't really matter much I don't think.
Contributor
Author
|
@karencfv Thanks for the review - this is ready for a second pass. I took several of your suggestions and left others out; here's example output from a racklette where we see several nonzero values from BGP settings being applied: and here's a steady-state output later when the reconciler has nothing to do, so we collapse the counts down to |
karencfv
reviewed
Aug 14, 2026
| ScrimletReconcilersStatus::WaitingForSledAgentNetworkingInfo => { | ||
| write!( | ||
| f, | ||
| "not running: sled-agent has not yet initialized \ |
Contributor
There was a problem hiding this comment.
Then how about changing the sled-agent has not yet initialized bit then? It implies that it will start running eventually no?
Comment on lines
+835
to
+847
| writeln!(f, "routes unchanged: {unchanged}")?; | ||
|
|
||
| if let (&Ok(deleted), &Ok(added)) = | ||
| (delete_v4_result, add_v4_result) | ||
| { | ||
| if deleted == 0 && added == 0 { | ||
| writeln!(f, "v4 routes: unchanged")?; | ||
| } else { | ||
| writeln!(f, "v4 routes:")?; | ||
| let mut f = IndentWriter::new(INDENT, &mut f); | ||
| writeln!(f, "deleted: {deleted}")?; | ||
| writeln!(f, "added: {added}")?; | ||
| } |
Contributor
There was a problem hiding this comment.
This bit is a little confusing to me. In the sample output below it says there are routes unchanged: 0 which I assume means something changed, but then it says all v4 and v6 routes are unchanged. So what changed then?
routes unchanged: 0
v4 routes: unchanged
v6 routes: unchanged
Contributor
Author
There was a problem hiding this comment.
Yeah that output is confusing, but it's technically accurate. 😅
So what changed then?
Nothing at all! The config for berlin when this ran was using only BGP, no static routes, so all three of those lines are correct: the reconciler did not leave any routes unchanged, and also didn't add or remove any v4 or v6 routes.
Couple ideas for making this clearer:
- Reword things slightly, something like:
routes left unchanged: 0
v4 routes modified: none
v6 routes modified: none
- Remove the "all 0" special case:
routes unchanged: 0
v4 routes:
deleted: 0
added: 0
v6 routes:
deleted: 0
added: 0
I mildly prefer the first one I think?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a lot smaller than it looks: ~2400 lines are changes to the lockstep API document, and another ~800 lines are
Displayimpls insideomdb. TheDisplayimpls are worth a quick review but aren't load bearing; I'll put example output from a racklette in a comment below.