Bugfix/ccip 11890 executor OOF metrics - #109
Merged
Merged
Conversation
Contributor
|
👋 kirtan-amin, thanks for creating this pull request! To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team. Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks! |
kirtan-amin
marked this pull request as draft
July 7, 2026 12:57
Contributor
|
Contributor
|
kirtan-amin
changed the title
Counter labeled {chainID, fromAddress}, emitted as both Prometheus and
OTel instruments, so operators can alert on a specific underfunded
sending address.
…tion Broadcaster and confirmer now call IncrementNumInsufficientFundsTxs with the sending address when a node rejects a tx for insufficient funds. The broadcaster also logs a meta-enriched warn, which carries messageID for transactions that set TxMeta.MessageIDs (e.g. CCIP).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…senderAddress label Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s warn The package's existing logs (confirmer, resender, txmgr) all use the fromAddress key for the sending address; keep the log stream on one vocabulary. The metric label remains senderAddress.
Code identifiers follow the package's fromAddress vocabulary; the externally-visible metric label stays senderAddress.
kirtan-amin
force-pushed
the
bugfix/CCIP-11890-executor-OOF-metrics
branch
from
July 7, 2026 14:03
eeab9e0 to
204cc36
Compare
kirtan-amin
marked this pull request as ready for review
July 8, 2026 08:30
amit-momin
approved these changes
Jul 8, 2026
yashnevatia
approved these changes
Jul 8, 2026
kirtan-amin
added a commit
to smartcontractkit/chainlink
that referenced
this pull request
Jul 13, 2026
…ent-funds observability
- chainlink-framework/{chains,metrics} -> 78ab531: TXM emits
tx_manager_insufficient_funds_tx_count{chainID,senderAddress} and a
retry warn log when an RPC rejects a tx for insufficient funds
(smartcontractkit/chainlink-framework#109)
- chainlink-ccv -> 690f2037: CL-mode executor transmitter attaches the
CCIP messageID to TxMeta so the TXM warn logs carry messageID
(smartcontractkit/chainlink-ccv#1242)
Covers CL-mode CCV executors and the keystone writer flow (CCIP-11890).
cyh7789
pushed a commit
to cyh7789/chainlink
that referenced
this pull request
Jul 17, 2026
* chore(deps): bump chainlink-framework and chainlink-ccv for insufficient-funds observability
- chainlink-framework/{chains,metrics} -> 78ab531: TXM emits
tx_manager_insufficient_funds_tx_count{chainID,senderAddress} and a
retry warn log when an RPC rejects a tx for insufficient funds
(smartcontractkit/chainlink-framework#109)
- chainlink-ccv -> 690f2037: CL-mode executor transmitter attaches the
CCIP messageID to TxMeta so the TXM warn logs carry messageID
(smartcontractkit/chainlink-ccv#1242)
Covers CL-mode CCV executors and the keystone writer flow (CCIP-11890).
* chore: bump framework (metric + chains) 527495d
* chore: bump framework (metrics + chains) to include OOF metrics
chore: remove changeset
chore: tidy submodule go.mod/go.sum for framework smartcontractkit#110 bump
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new metric that tracks when we get an error on broadcasting for insufficient funds at the sender address. This metric will surface the case were an executor address needs topping up so we can act to notify the executor to reduce the number of txs that get stuck and the time they are stuck for
CCIP-11890
Requires Dependencies
Resolves Dependencies