Queue Processor deletes an SQS message after a taint error

[enumura1]

Problem

When Queue Processor handles an EC2 Instance Rebalance Recommendation, a failure to add a taint to the Node is not treated as an error.

The taint error is logged, but node draining and SQS message deletion continue.
The SQS message cannot be retried.

Related

#1279 updated the Queue Processor Spot interruption handler to return taint errors so that the SQS message can be retried.

The EC2 Instance Rebalance Recommendation handler does not propagate the error in the same way.

Steps to reproduce

1. Set up NTH in Queue Processor mode, two EKS Nodes, an SQS queue, and a test Pod.
2. Configure a ValidatingAdmissionPolicy that rejects taint updates by NTH for the target Node.
3. Send an EC2 Instance Rebalance Recommendation event for the target Node directly to the SQS queue.
4. Check the NTH logs, target Node, test Pod, and SQS message.

No EC2 instance was terminated. The event was sent directly to the test SQS queue in the EC2 EventBridge event format.

Expected outcome

When adding the taint fails, Queue Processor should stop processing the event.

• The Node should not be cordoned or drained.
• The SQS message should not be deleted.
• The same SQS message should be received again after the visibility timeout.

Actual outcome

• The taint update was rejected.
• Node draining continued.
• The test Pod moved to the other Node.
• The SQS message could not be received again after a 30-second visibility timeout.

Application Logs

  Unable to taint node with taint
  aws-node-termination-handler/rebalance-recommendation

  Draining the node node_name=<target-node>

  evicting pod nth-verification/test-app

Environment

• NTH revision: 62a8761 (built locally)
• NTH mode: Queue Processor
• Worker Node OS / architecture: Amazon Linux 2023 / amd64
• Kubernetes version: EKS 1.34
• Installation method: Helm chart from the same revision
• SQS visibility timeout: 30 seconds