Repository navigation
fix(polling): drop a recovered device's failed row once its notice is sent (#737) - #871
Merged
bmfmancini merged 2 commits intoNov 6, 2026
Merged
Conversation
…ti#737) thold_update_host_status() announces every device in plugin_thold_host_failed that is up again, but the table was only cleared and rebuilt after all recovered devices had been handled. Each device is handled synchronously (up to four SNMP gets, a blocking exec() of the up command, and mail), so with many recoveries the run can outlast the poller cycle. When it was cut short before the rebuild, every row stayed and the next cycle sent the same recovery notices again. Delete each recovered device's row, by its row id, as soon as its recovery has been handled. The end-of-run rebuild is unchanged and still records devices that are down or recovering. The in-memory $failed list it consults for recovering devices is unaffected.
…acti#737) The first change deleted a recovered device's row only after all of its work, including the blocking up command. Validating on a real poller showed a run cut short inside that command still repeated the device's recovery notice next cycle, which is the case Cacti#737 describes. Delete the row right after the recovery notice is sent, before the up command. A run that dies in the command no longer repeats the notice; a run that dies before the notice was sent (for example in the SNMP gets) keeps the row, so the notice is sent on the next cycle instead. Adds a test for a run cut short inside the up command; it fails with the previous placement.
|
somethingwithproof
approved these changes
Nov 6, 2026
somethingwithproof
left a comment
Member
There was a problem hiding this comment.
No blockers found. The full suite passes on Linux/PHP 8.4; PHP 8.2–8.4 CI is green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Description
thold_update_host_status()announces every device inplugin_thold_host_failedthat's up again. The table was only cleared and rebuilt after all recovered devices had been handled.Each device is handled synchronously: up to four SNMP gets, mail, and a blocking
exec()of the device up command. With several recoveries, the run can outlast the poller cycle. When it was cut short before the rebuild, every row stayed, and the next cycle sent the same recovery notices again.This PR deletes each recovered device's row, by row id, right after its recovery notice is sent and before the up command:
The end-of-run rebuild is unchanged.
The branch has two commits. The first deleted the row after the up command, and server testing showed that still repeated the notice when the command was the step that ran long. The second moves the delete to before the command. Squash if you prefer.
Related Issue
Fixes #737.
Motivation and Context
On larger installs, a burst of recoveries makes thold re-send the same "returned from DOWN state" mail every cycle until a run finally completes.
How Has This Been Tested?
Unit tests (Pest): a new
RecoveryNoticeRepeatTestcovers three cases:All three fail on
develop. The up-command case also fails with the first commit alone, and passes with both commits.Real poller: Cacti 1.2.32, PHP 8.4, cactid every 60 s, with each commit deployed in turn.
127.0.0.1so they recovered.poller_thold.phpprocess on the first recovery, standing in for a run that outlasts its cycle.The settings changed for the test (
alert_deadnotify,alert_email,thold_device_command) were restored afterwards, and the devices removed.Unrelated, seen while testing: every device-down notice logs
Undefined variable $last_time_durationatincludes/polling.php. The down command'sTHOLD_DOWNTIMEuses$last_time_durationwhere the variable is$last_state_duration. I'll send that as a separate PR.Types of changes
Checklist: