Summary
Fixing the wrong-health report surfaced that the x-medkit-graph health signal
is not robust and gives the operator no controls. This tracks the enhancement
work that goes beyond the bug fix itself.
Gaps:
- Freshness age was computed on a wall clock, so a backward clock step (for
example an NTP correction on a robot that boots before NTP sync) could hide a
dead topic as fresh.
- A single late
/diagnostics sample (DDS or executor jitter) could flip the
pipeline to broken and back, with no debounce.
frame_rate_msg is a topic-level rate summed over all publishers on a topic.
A duplicate or leftover publisher inflates it, so a slow producer can read as
healthy, and nothing told the operator the rate was suspect.
- There was no way to tune any of this per deployment.
Proposed solution
- Compute freshness age on a monotonic clock; keep the human-readable timestamp
on wall time.
- Add
stale_grace_sec (per-plugin and per-function): report metrics_stale
only after the age has been past the window continuously for the grace, so a
single late sample does not flap. 0 keeps the old behaviour.
- Report
publisher_count and rate_ambiguous on each edge, and add
multi_publisher_rate (annotate / suppress) so a deployment can choose to
keep the flagged rate or hide the untrustworthy number.
- Cover the behaviour with integration tests that drive a real diagnostics
producer through the gateway.
Additional context
Builds on the wrong-health bug fix (#545). Both are addressed in the same pull
request.
Summary
Fixing the wrong-health report surfaced that the
x-medkit-graphhealth signalis not robust and gives the operator no controls. This tracks the enhancement
work that goes beyond the bug fix itself.
Gaps:
example an NTP correction on a robot that boots before NTP sync) could hide a
dead topic as fresh.
/diagnosticssample (DDS or executor jitter) could flip thepipeline to
brokenand back, with no debounce.frame_rate_msgis a topic-level rate summed over all publishers on a topic.A duplicate or leftover publisher inflates it, so a slow producer can read as
healthy, and nothing told the operator the rate was suspect.
Proposed solution
on wall time.
stale_grace_sec(per-plugin and per-function): reportmetrics_staleonly after the age has been past the window continuously for the grace, so a
single late sample does not flap.
0keeps the old behaviour.publisher_countandrate_ambiguouson each edge, and addmulti_publisher_rate(annotate/suppress) so a deployment can choose tokeep the flagged rate or hide the untrustworthy number.
producer through the gateway.
Additional context
Builds on the wrong-health bug fix (#545). Both are addressed in the same pull
request.