Skip to content

fix(polling): drop a recovered device's failed row once its notice is sent (#737) - #871

Merged
bmfmancini merged 2 commits into
Cacti:developfrom
bmfmancini:fix/737-recovery-notice-repeat
Nov 6, 2026
Merged

bmfmancini merged 2 commits into
Cacti:developfrom
bmfmancini:fix/737-recovery-notice-repeat

Conversation

@bmfmancini

Copy link
Copy Markdown
Member

Description

thold_update_host_status() announces every device in plugin_thold_host_failed that's up again. The table was only cleared and rebuilt after all recovered devices had been handled.

Each device is handled synchronously: up to four SNMP gets, mail, and a blocking exec() of the device up command. With several recoveries, the run can outlast the poller cycle. When it was cut short before the rebuild, every row stayed, and the next cycle sent the same recovery notices again.

This PR deletes each recovered device's row, by row id, right after its recovery notice is sent and before the up command:

  • A run that dies inside the command no longer repeats the notice.
  • A run that dies before the notice was sent (for example in the SNMP gets) keeps the row, so the notice is sent next cycle instead of being lost.

The end-of-run rebuild is unchanged.

The branch has two commits. The first deleted the row after the up command, and server testing showed that still repeated the notice when the command was the step that ran long. The second moves the delete to before the command. Squash if you prefer.

Related Issue

Fixes #737.

Motivation and Context

On larger installs, a burst of recoveries makes thold re-send the same "returned from DOWN state" mail every cycle until a run finally completes.

How Has This Been Tested?

Unit tests (Pest): a new RecoveryNoticeRepeatTest covers three cases:

  • the run dies at the next device;
  • the run dies inside the first device's up command;
  • a full run clears each row.

All three fail on develop. The up-command case also fails with the first commit alone, and passes with both commits.

Real poller: Cacti 1.2.32, PHP 8.4, cactid every 60 s, with each commit deployed in turn.

  1. Two test devices on an unreachable address went down.
  2. They were pointed at 127.0.0.1 so they recovered.
  3. The device up command killed the poller_thold.php process on the first recovery, standing in for a run that outlasts its cycle.
Recovery mails develop first commit only this PR
Device 1 (run killed in its up command) 2 2 1
Device 2 1 1 1

The settings changed for the test (alert_deadnotify, alert_email, thold_device_command) were restored afterwards, and the devices removed.

Unrelated, seen while testing: every device-down notice logs Undefined variable $last_time_duration at includes/polling.php. The down command's THOLD_DOWNTIME uses $last_time_duration where the variable is $last_state_duration. I'll send that as a separate PR.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation repository accordingly.

…ti#737)

thold_update_host_status() announces every device in
plugin_thold_host_failed that is up again, but the table was only
cleared and rebuilt after all recovered devices had been handled. Each
device is handled synchronously (up to four SNMP gets, a blocking exec()
of the up command, and mail), so with many recoveries the run can
outlast the poller cycle. When it was cut short before the rebuild,
every row stayed and the next cycle sent the same recovery notices
again.

Delete each recovered device's row, by its row id, as soon as its
recovery has been handled. The end-of-run rebuild is unchanged and
still records devices that are down or recovering. The in-memory
$failed list it consults for recovering devices is unaffected.
…acti#737)

The first change deleted a recovered device's row only after all of its
work, including the blocking up command. Validating on a real poller
showed a run cut short inside that command still repeated the device's
recovery notice next cycle, which is the case Cacti#737 describes.

Delete the row right after the recovery notice is sent, before the up
command. A run that dies in the command no longer repeats the notice; a
run that dies before the notice was sent (for example in the SNMP gets)
keeps the row, so the notice is sent on the next cycle instead.

Adds a test for a run cut short inside the up command; it fails with the
previous placement.
Copilot AI balanced review requested due to automatic review settings November 6, 2026 06:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@sonarqubecloud

sonarqubecloud Bot commented Nov 6, 2026

Copy link
Copy Markdown

@somethingwithproof somethingwithproof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers found. The full suite passes on Linux/PHP 8.4; PHP 8.2–8.4 CI is green.

@bmfmancini
bmfmancini merged commit 01fd198 into Cacti:develop Nov 6, 2026
5 checks passed
@bmfmancini
bmfmancini deleted the fix/737-recovery-notice-repeat branch November 6, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

thold does not clear plugin_thold_host_failed when it doesn't have time to finish, causing a loop

3 participants