Skip to content

Robustness and operator controls for x-medkit-graph health #547

Description

@bburda

Summary

Fixing the wrong-health report surfaced that the x-medkit-graph health signal
is not robust and gives the operator no controls. This tracks the enhancement
work that goes beyond the bug fix itself.

Gaps:

  • Freshness age was computed on a wall clock, so a backward clock step (for
    example an NTP correction on a robot that boots before NTP sync) could hide a
    dead topic as fresh.
  • A single late /diagnostics sample (DDS or executor jitter) could flip the
    pipeline to broken and back, with no debounce.
  • frame_rate_msg is a topic-level rate summed over all publishers on a topic.
    A duplicate or leftover publisher inflates it, so a slow producer can read as
    healthy, and nothing told the operator the rate was suspect.
  • There was no way to tune any of this per deployment.

Proposed solution

  • Compute freshness age on a monotonic clock; keep the human-readable timestamp
    on wall time.
  • Add stale_grace_sec (per-plugin and per-function): report metrics_stale
    only after the age has been past the window continuously for the grace, so a
    single late sample does not flap. 0 keeps the old behaviour.
  • Report publisher_count and rate_ambiguous on each edge, and add
    multi_publisher_rate (annotate / suppress) so a deployment can choose to
    keep the flagged rate or hide the untrustworthy number.
  • Cover the behaviour with integration tests that drive a real diagnostics
    producer through the gateway.

Additional context

Builds on the wrong-health bug fix (#545). Both are addressed in the same pull
request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions