Warn about metrics whose values don't match their gauge/counter type

[eastagiletracker]

This PR proposes making the agent warn about metrics whose collected values don't match the gauge/counter type they were defined with (Fixes #32). We include this PR work along with a full history of your repo at https://eastagiletracker.com/projects/308. You can sign in with your GitHub ID to claim ownership of the project.

What changed and why

A metric defined as gauge that is really cumulative (or a counter that is really a point-in-time value) is easy to write into a new integration YAML and very hard to notice afterwards, because the agent collects and ships it without complaint. The agent is, however, in the one position to notice: it sees every value the source produces.

MetricTypeMismatchDetector (new, in spm-monitor) watches the values as they are read from the source — inside AttributeObservation.getMetricValue(), before counter values are turned into deltas, so the observation is made at the source and not in the agent's output — and logs a warning when a metric defined as a gauge only ever grew, or when a metric defined as a counter kept dropping. A metric is reported only after at least 30 of its values were collected, and a counter is reported only when the drops make up at least 1/10 of its collected values, so a service restart resetting a real counter stays quiet. Every metric is reported at most once. Because the tracking keeps per-metric state and runs on the collection path, it is enabled only when the agent runs with SPM_MONITOR_LOGGING_LEVEL=DEBUG, and the number of tracked metrics is capped at 5000. On any other logging level the added work is a single boolean read per collected value and behavior is unchanged. docs/how-to.md describes how to turn it on and what the warning looks like.

The hook sits on the shared value path of every data source type: getMetricValue() is what JsonAttributeObservation, DbAttributeObservation, MBeanAttributeObservation and MBeanFunctionInvoker all call, so JSON, DB and JMX metrics are covered by the one change.

Reproduced on master at 4d68b2c (Java 8, mvn -B test): collecting a JSON metric declared type: gauge whose source value only grows (10, 20, 30, …) 60 times through JsonObservation.collectStats(), with the agent logging on DEBUG, produces an agent log of exactly 0 characters — the misconfiguration is completely silent today.

Verification: mvn -B test was run on the clean tree before the change and again after it — same result, no new failures, and spm-monitor goes from 89 to 103 tests. The new tests are MetricTypeMismatchDetectorTest (thresholds, counter resets, fluctuating gauges, non-numeric and NaN values, one-report-per-metric, the tracking cap) and MetricTypeMismatchCollectionTest, which drives real JsonObservation.collectStats() collections and asserts on the agent log. Removing only the two-line call in getMetricValue() and re-running turns testGrowingGaugeIsReportedWhileCollecting red (expected:<1> but was:<0>) while the "correctly declared counter", "fluctuating gauge" and "detector disabled" controls stay green, so the tests really do measure the new behavior.

Nothing existing changed shape: no public signature was modified, no default was altered, no dependency was touched. MetricTypeMismatchDetector.setEnabled(...) is deliberately public so the state can be forced in tests; its default comes from the agent's own logging level. The thresholds are constructor parameters if you would rather tune them than take the defaults.

How this was managed

This work was tracked as Make agent warn about possibly misconfigured gauge/counter on the board at https://eastagiletracker.com/projects/308, which was imported from this repository's issues and pull requests (151 stories) and used to manage the change.

If you'd rather not receive contributions like this, reply no-more-prs on this pull request and we won't open any further ones on your repositories.

---

Lawrence W. Sinclair
CEO / East Agile
linkedin.com/in/lwsinclair/
eastagile.com