Skip to content

Feature: Repeated sending for DIRECT-route packets with echo-cancellation (improves delivery, reduces floods) - #2670

Open
usrflo wants to merge 5 commits into
meshcore-dev:devfrom
usrflo:feature/repeated-sending-2
Open

usrflo wants to merge 5 commits into
meshcore-dev:devfrom
usrflo:feature/repeated-sending-2

Conversation

@usrflo

@usrflo usrflo commented Jun 2, 2026 •

Copy link
Copy Markdown

Relates to #1342 (comment).

Motivation

Direct messages over multi-hop paths fail silently when a single hop misses a packet due to temporary radio interference or channel congestion. The sending node then falls back to a flood — which is both unreliable under load and consumes significant airtime. In my tests most incoming paths with a length of 3 hops couldn't be reused in back traces: a far too low number that should be increased by an error correction.

What this PR does

This branch adds conditional repeated sending for DIRECT-route packets at the repeater level:

  • After forwarding a DIRECT packet, a repeater schedules up to max_resend_attempts re-transmissions (default: 2, configurable 0–3 via set max.resend).
  • Each pending retransmit is cancelled immediately when the repeater overhears the downstream relay successfully forwarding the same packet (echo-detection via a cached packet hash in the outbound queue). The same cancel scan runs for TRACE forwards inside the TRACE branch (see below).
  • The cancel window is sized to the packet: W = C0 + K·airtime + margin + (attempt−1)·jitter under an airtime-proportional ceiling, so it always outlasts the downstream forward at any length or spreading factor, without padding short packets.
  • Resends use a non-invasive channel gate (isResendChannelActive(): preamble-detect IRQ + RSSI margin over the noise floor, no CAD) so the radio stays in RX and can still overhear the cancelling forward. First sends keep the normal CAD carrier sense.
  • The final relay hop of TXT_MSG gets exactly one ACK-cancellable resend (final_hop_ack_resend); the returning destination ACK cancels it (cancelPendingFinalHopResend).
  • ACK packets use a dedicated deduplication table (_acks[] in SimpleMeshTables) to keep deduplication cheap (4-byte compare, no SHA-256) and to prevent ACK entries from evicting long-lived flood-packet hashes.
  • Pool-shedding: under low free-pool pressure no resend is queued at all — the redundant retry is dropped before it can compete with RX for pool slots.

TRACE handling

Two TRACE issues in the resend logic were found by @Kernel-Error and reviewed/measured by @m0urs on a live repeater (m0urs#25, m0urs#26) — both are fixed here:

  1. Queued resends of TRACE forwards were never cancelled. The TRACE branch in Mesh::onRecvPacket() always returns, so the generic echo-cancel block was never reached for TRACE and every intermediate hop re-sent each TRACE up to max_resend_attempts times even though the next hop had already forwarded it. The TRACE branch now performs its own cancel scan (payload/SNR-prefix match via isRetryMatch(), well-formed non-terminal legs only).
  2. The last-hop TRACE forward pre-set sending_attempts = max to suppress retries, which made the first (and only) send look like a resend: it went through the resend LBT gate instead of the normal channel check and was counted in the resend statistics. Replaced by a runtime-only no_retry flag on Packet.

Sim A/B for the TRACE fixes (mcsim, trace_retry topology, 3 hops of ~87 % marginal links, 8 TRACEs per run, 3 seeds):

TX total (Alice+R1+R2+R3) Delivered
without TRACE fixes 188 23/24
with TRACE fixes 102 (−46 %) 24/24

Comparison with PR #2367 (HALO)

This implementation shares the same goal as #2367 — improve direct-path reliability — but uses a different strategy. Both branches were applied to mcsim (including fixes to make the simulator timing-accurate for retransmit scenarios, see simulator commits) and run against the same standardised test scenario.

Test topology: 3 repeaters between Alice and Bob, all links of marginal/bad quality
Test: 10 random seeds × 20 DMs each = 200 deliveries total

Branch Avg delivered Avg TX (direct) Avg collisions
halo-direct-path-retries (#2367) ~15.9 / 20 ~420 ~15
repeated-sending-2 (this PR) ~19.4 / 20 ~187 ~3

Key advantages of this approach:

  • Higher delivery rate (~97 % vs ~80 % in the test scenario)
  • 55 % fewer TX packets — echo-cancellation prevents unnecessary retransmits before they fire
  • ~80 % fewer collisions — significantly less channel saturation
  • Simpler and more resource-efficient — no SNR-gated retry logic, no neighbor-table lookup; just overhear and cancel

Delivery rate vs. resend attempts (mcsim, retry_showcase topology)

max_resend_attempts Delivery rate
0 (stock MeshCore) 71 %
1 91 %
2 (default) 95.5 %
3 97 %

A value of 2–3 is a good compromise between reliable delivery and channel efficiency.

Relationship to PR #2933 (noise-floor estimator)

This PR intentionally contains no radio-driver changes: isResendChannelActive() calls the existing getNoiseFloor() interface, which today is the one-way ratchet. PR #2933 replaces that estimator with a median-based one; both PRs are complementary and can be merged independently. With #2933 merged, the 12 dB resend margin works against a floor that does not drift to −120 dBm under load (on hardware where the ratchet drifts, the resend gate becomes overly conservative over time — the fixed gate matters more the longer a node runs).

CLI

No new commands are needed — max_resend_attempts is already exposed:

> get max.resend
> 2, resends 64/173 (37%)

@usrflo

usrflo commented Jun 2, 2026

Copy link
Copy Markdown
Author

Some information about the 3 commits:

  1. the first commit MESHCORE_SIMULATOR patch contains common changes to use "mcsim" (see https://github.com/Brent-A/mcsim and https://github.com/usrflo/mcsim) or any other simulation that requires simulation hooks. This commit is not required for this feature but is a requirement to run simulated comparisons.
  2. the second commit Implemented repeated sending for error correction contains changes to the MeshCore implementation (dev branch)
  3. the third commit Added comments to explain changes related to the repeated-sending feature is meant for the code review. This commit might be removed if this pull request is accepted.

@cwichura

cwichura commented Jun 3, 2026

Copy link
Copy Markdown

This seems very similar to #2367.

@usrflo

usrflo commented Jun 12, 2026 •

Copy link
Copy Markdown
Author

After mcsim-simulation runs and internal tests on real hardware I released firmware packages named "ufo_0.1" for repeaters, companions and room-servers at https://github.com/usrflo/MeshCore/releases. This firmware is based on MeshCore-dev v1.16.0 with application of this pull request.

It would be great to get some feedback on this feature. @wlockwood?

@usrflo

usrflo commented Jun 19, 2026

Copy link
Copy Markdown
Author

To recognize propagated TRACE packages a special detection was required as TRACE packages differ in using path and body fields. In the simulation with a relatively stable chain of repeaters (TOPOLOGY, BEHAVIOR) I got the following results:

Without repeated sending (max.resend 0)
averages over 10 seeds:
sent per run: 8.0
received total per run: 6.2
collisions per run: 0.0
overall delivery rate: 77.5%

With repeated sending (max.resend 2)
averages over 10 seeds:
sent per run: 8.0
received total per run: 8.0
collisions per run: 3.6
overall delivery rate: 100.0%

@usrflo
usrflo marked this pull request as draft June 26, 2026 10:58
@usrflo
usrflo marked this pull request as ready for review June 26, 2026 20:57
@usrflo
usrflo marked this pull request as draft June 30, 2026 19:17
@usrflo
usrflo marked this pull request as ready for review July 11, 2026 12:33
@usrflo

usrflo commented Jul 11, 2026 •

Copy link
Copy Markdown
Author

After the last bugfix I tested the firmware on the following hardware device chain:
C1 -> R1 -> R2 -> R3 -> C2 with weak links (-80 / -90 / -100 RSSI) between each of these nodes (strongly isolated devices with weak antennas). A successful delivery was counted when C2 sent out an ACK.

Without repeated sending: 81 of 100 packages were delivered
With repeated sending: 96 of 100 packages were delivered 🚀

Side note:
These tests were executed with

  1. the noise_floor calculation fix from https://github.com/usrflo/MeshCore/tree/fix/noise-floor-ratchet, see Fix/noise floor ratchet, adapt noise to the real floor with recovery in both directions #2933
  2. the non blocking waiting for silence before TX from https://github.com/usrflo/MeshCore/tree/feature/quiet-dwell, see README-quiet-dwell.md

See the according firmware release ufo_0.5 at https://github.com/usrflo/MeshCore/releases

diagram-resend chart-delivery

To give an idea about the measurement: I used 2 observers to check the details on the two separated locations.
The SNR/RSSI values in the following excerpt are those of the observer. The DUP at the end of the line marks a repeated package sending.

One sample:

13:03:24.523  Δ=  7.508s  Direct·TXT_MSG sender=MeshL1New origin=MeshL1New dest=WL1_C2  path=ND1_R1→BB1_R4→SB1_R2  hop 1/4  SNR= +3.2 RSSI= -72  871CD76C34937AEC ⚠noi
┃ ⚠ INTERFERENCE END    noise      13:03:18  dur=6.5s peak=+40  pkts=0 rx_errs=0
13:03:25.299  Δ=  0.776s  Direct·TXT_MSG sender=ND1_R1 origin=MeshL1New dest=WL1_C2  path=BB1_R4→SB1_R2  hop 2/4  SNR= +9.5 RSSI= -59  871CD76C34937AEC
13:03:26.329  Δ=  1.030s  Direct·TXT_MSG sender=ND1_R1 origin=MeshL1New dest=WL1_C2  path=BB1_R4→SB1_R2  hop 2/4  SNR=+13.2 RSSI= -59  871CD76C34937AEC DUP ⟳1.030s
13:03:26.847  Δ=  0.518s  Direct·TXT_MSG sender=BB1_R4 origin=MeshL1New dest=WL1_C2  path=SB1_R2  hop 3/4  SNR=+13.2 RSSI= -75  871CD76C34937AEC
13:03:27.398  Δ=  0.551s  Direct·TXT_MSG sender=SB1_R2 origin=MeshL1New dest=WL1_C2  path=∅  hop 4/4  SNR=+12.0 RSSI= -49  871CD76C34937AEC
13:03:27.915  Δ=  0.517s  Direct·ACK   sender=direct    path=∅  hop 1/1  SNR=+11.8 RSSI= -76  70E804018D91907B
┃ ⚠ INTERFERENCE START  contention 13:03:33  nf=-108 rssi=-76 margin=+32 (dev_ms=8340913)
  └─ chain MeshL1New → ND1_R1 → BB1_R4 → SB1_R2 → WL1_C2  | 6 RX | SNR +3.2..+13.2 RSSI -76..-49 | TXT_MSG

@usrflo

usrflo commented Jul 18, 2026

Copy link
Copy Markdown
Author

Today's merges were a setback; I will try to find the reason. So long this pull request is set back to draft.

@usrflo
usrflo marked this pull request as draft July 18, 2026 21:22
@usrflo

usrflo commented Jul 19, 2026

Copy link
Copy Markdown
Author

Good numbers again, I fixed the timing issues of the last commits.

The following problems are addressed in the changes of the last days:

  • schedule the sending after noise detection to 3 different wait buckets based on the residual path length (mod 3) of the direct message; so multiple nodes nearby don't send at the same time after noise detection
  • don't resend packages if the sending queue becomes filled up too much (prevent congestion)
  • appropriate timing for package cancellations before a resend. This depends on the airtime and constant wait intervals for the processing on the nodes.

Last results with the test chain Companion1->Rep1->Rep2->Rep3->Companion2, 50 direct messages in a 10 second interval, up to 2 package repetitions (default). Every ACK is counted as success:
a) sending over very weak links (-70 ... -100 RSSI) with an interferer node (an advert every 6 seconds): 46 ACKs
b) sending over very weak links (-70 ... -100 RSSI) without an interferer node: 49 ACKs
c) sending over stable links: 50 ACKs

@usrflo
usrflo marked this pull request as ready for review July 19, 2026 13:35
@usrflo
usrflo force-pushed the feature/repeated-sending-2 branch from 81e7fe2 to 8acdb11 Compare July 21, 2026 14:38
@usrflo

usrflo commented Jul 21, 2026 •

Copy link
Copy Markdown
Author

Last update: the max wait time limit - used as a window to cancel packages prepared to be resended - needs to be calculated according to the airtime / package size.

@usrflo

usrflo commented Jul 22, 2026 •

Copy link
Copy Markdown
Author

Repeated-sending - current state and the safeguards that keep it stable

A resend is a redundant retransmission of a DIRECT (acknowledged-path) packet, queued by the originator and auto-cancelled when a downstream relay's forward — or the destination's ACK — is overheard. The logic only ever adds a retry that can be called off, and sheds/collisions are guarded as follows.

Safeguards

  • Scoped eligibility + attempt cap. Only direct-routed packets still carrying ≥1 relay hash are resendable (there must be a downstream forward that can cancel it). The final hop has no downstream forward, so it gets exactly one resend, cancelled by the returning ACK. Retries are capped at getMaxResendAttempts() (default 2).

  • Non-invasive resend LBT. Resends deliberately skip CAD so the radio stays in RX and can still overhear the cancelling forward. Instead isResendChannelActive() gates on isReceivingPacket() || (currentRSSI − noiseFloor ≥ RESEND_INTERFERENCE_MARGIN), margin fixed at 12 dB (as long as the prefs int.threshold discussion goes on). The RSSI-energy term detects an ongoing transmission at any point — preamble or payload — so a long packet is not overlaid by a retry. First sends keep normal CAD carrier-sense, but it's recommended to set cad off as it seems to increase deafness.

  • Cancel window sized to packet length / SF. The wait before a resend fires is C0 + K·airtime + margin + (attempt−1)·jitter, under an airtime-proportional ceiling (CAP_BASE + CAP_AIRTIME_X·airtime). The window therefore always outlasts the downstream forward for any length or spreading factor. The earlier flat 1500 ms cap under-covered SF10 long messages, so every long packet fired its resend before the forward could cancel it — a redundant, colliding resend on each one.

  • Per-hop busy-recheck bucket (resenders stay out of each other's way). When the channel is busy, deferred resenders retry at (pathHashCount % 3 + 1) × 120 ms instead of all firing on the same loop tick. The hash count drops by exactly 1 per forwarding hop, so any two mutually-in-range chain neighbours (≤2 hops apart) always land in different buckets and never collide on retry; same-bucket pairs are ≥3 hops apart and don't interfere. This is a re-check offset only — the cancel window is not inflated.

  • Pool-shedding (queue-overflow protection). A resend shares a packet-pool slot with RX, so it is only queued when the free pool is healthy (freeCount > 6). Under sustained load the redundant retry is dropped rather than exhausting the pool and deafening the node — the primary direct forward has already been transmitted.


If you like to test this PR it's recommended to use:

set cad off
set agc.reset.interval 0     # I only tested with 0, so better to do so
set max.resend 2

This PR ist part of the ufo firmware, latest build at time of writing: v0.6

@usrflo

usrflo commented Jul 27, 2026

Copy link
Copy Markdown
Author

With the latest checkin statistics on package resending can be viewed, sample:

> get max.resend
> 2, resends 64/173 (36%)

@m0urs

m0urs commented Aug 9, 2026

Copy link
Copy Markdown

I have been testing this PR (and their modifications) since a few weeks now, and I confirm that this is a HUGE improvement.

I have two recipients, both 4 Hops away and all 4 repeaters have this PR installed. Sind I have that PR in place, I do have a 100% delivery and ACK rate to both recipients (and vice versa). I never had that before with the standard firmware without that modification.

So I really would like to have this PR included in the next official release.

Thanks @usrflo for that great improvement!

@Alain2019

Copy link
Copy Markdown

Are there reasons to keep the conflicts unsolved?

I see this as a huge improvement to meshcore.

@wlockwood

Copy link
Copy Markdown

Are there reasons to keep the conflicts unsolved?

I see this as a huge improvement to meshcore.

There's a competing approach by mikecarper that solves similar problems in a slightly different way: https://github.com/mikecarper/MeshCore/releases/tag/v1.17.1.5-halo-keymind-cascade-dev-26303793

@Alain2019

Copy link
Copy Markdown

Are there reasons to keep the conflicts unsolved?
I see this as a huge improvement to meshcore.

There's a competing approach by mikecarper that solves similar problems in a slightly different way: https://github.com/mikecarper/MeshCore/releases/tag/v1.17.1.5-halo-keymind-cascade-dev-26303793

Not much info about the mike carper version. As I understand it it's a simple repeat if not heard. This should be used with quite some care with flood messages. A repeater with 32 neighbours. Is it sufficient that only one resend it or all 32?

Is there some more info?

@wlockwood

Copy link
Copy Markdown

I'm not the developer, but there's a lot going on with HALO/Keymind:

  • DMs: Keymind doesn’t merely repeat unless it hears any duplicate. It listens for the intended downstream node forwarding the packet—effectively a hop-level ACK—and cancels retries only on confirmed path progress. The final relay sends one extra copy because the recipient produces no forwarding echo.

  • Floods: Keymind retries until it hears the packet from a qualifying downstream repeater, not just anyone. In bridge mode it can require echoes from every relevant direction/bucket. Upstream or ignored repeaters don’t prematurely cancel retries.

@Alain2019

Copy link
Copy Markdown

I'm not the developer, but there's a lot going on with HALO/Keymind:

* **DMs**: Keymind doesn’t merely repeat unless it hears any duplicate. It listens for the intended downstream node forwarding the packet—effectively a hop-level ACK—and cancels retries only on confirmed path progress. The final relay sends one extra copy because the recipient produces no forwarding echo.

* **Floods**: Keymind retries until it hears the packet from a qualifying downstream repeater, not just anyone. In bridge mode it can require echoes from every relevant direction/bucket. Upstream or ignored repeaters don’t prematurely cancel retries.

Seems also nice and also bigger than this PR. But it seems completely out of the official code tree. Also I didn't find any information about how it works.

Ideally the two should be merged or better the best of both should be merged into the official code tree.

@wlockwood

Copy link
Copy Markdown

Ideally, yes. I'd love to see retries of some flavor make it into the official firmware. 2 per-hop retries and 2 end-to-end retries makes a huge difference for the usable number of hops a client can communicate over:
image
In the above table, each cell indicates the number of hops a message can span while maintaining a delivery certainty of 99% assuming a link reliability along the top.

@Alain2019

Copy link
Copy Markdown

Ideally, yes. I'd love to see retries of some flavor make it into the official firmware. 2 per-hop retries and 2 end-to-end retries makes a huge difference for the usable number of hops a client can communicate over: image In the above table, each cell indicates the number of hops a message can span while maintaining a delivery certainty of 99% assuming a link reliability along the top.

Yes. There should be a rather good reliability for a rather low nr of hops, lets say 5-6 hops. This is now NOT case. It's simply not reliable enough to do really useful stuff with meshcore.

@wlockwood

Copy link
Copy Markdown

For routed messages, you're not wrong. Flood messages are fairly reliable in a dense network like we have in the PNW.

@usrflo

usrflo commented Sep 10, 2026

Copy link
Copy Markdown
Author

Conflict with current dev is resolved - the branch is up to date again.

Referring to HALO/Keymind: I tested that fork side by side (mcsim + hardware); my evaluation is here: #2367 (comment).

For DMs, HALO waits for the intended downstream forward - essentially the same principle used by this PR "repeated-sending": an implicit hop-level ACK, cancelled when the forward is overheard. I don't see a fundamentally better approach there. The gains also remained modest: in mikecarper's own 30-seed mcsim DM test, delivery improved from 150→153 and ACKs from 38→39.

Where I deliberately stopped with the repeated sending:

  • No retries on flood. Per-hop flood retries would turn every non-echoing listener into a retransmitter. The 32-neighbour amplification @Alain2019 sketches is real, and as @wlockwood points out, floods are already fairly reliable in dense networks. Repetition is much more useful where a lost packet has exactly one intended next hop - i.e. direct/routed messages.

  • Retries are bounded at 2. Long-running hardware tests on a lossy reference chain showed 2 retries to be the sweet spot. Beyond roughly 3 potential retries, packet collisions start to outweigh the gains. There is only one configuration knob: max.resend, defaulting to 2 and allowing 0–3. Keeping the configuration simple is a feature here, not a limitation.

  • Resend LBT. The resend LBT (12 dB margin above a median-estimated noise floor, without CAD) also came out of that hardware testing. The underlying noise-floor fix is tracked in Fix/noise floor ratchet, adapt noise to the real floor with recovery in both directions #2933. The RX-focused LBT approach maximizes the likelihood to hear a downstream package repetition while CAD blocked RX too much and decreased deliverability in my tests.

Recap: mcsim improved from 77.5%→100%; the hardware reference chain showed an improvement from 81/100→96/100; and ACKs reached 46–50/50 depending on interference.

If you see the requirement: I'm happy to merge the best of both approaches. What exactly do you see as an advantage of the HALO implementation?

@mikecarper

Copy link
Copy Markdown

No retries on flood. I limit this to hops 0-2; after that it doesn't try. This is the biggest win for user experience actually; like this is the main reason people install the firmware I put out from this branch https://github.com/mikecarper/MeshCore/tree/keymindCascade.

Retries are bounded at 2. We're dealing with a repeaters with 200+ neighbors and 2 tries doesn't work.

Resend LBT. I need to look into this as it might help the situation but the big repeaters can be locked up for 8+ seconds here (the queue depth stays + for the entire time)

@Alain2019

Copy link
Copy Markdown

As far as I can see it I think that @usrflo it's forwarding of DM is further tested and thought over.

I do see a benefit for flood's also, but with limited resends. If I read @mikecarper his comment it's limited to the first hops. That could be a solution, but maybe also the situation of a busy "air" (lot of traffic) and not a single resend. I see flood's failing on a few hops and that's not nice. I have also a few "high", busy and "strong" repeaters in the neighbourhood, maybe not the best situation.

Making meshcore more reliable is the most important.

@Alain2019

Copy link
Copy Markdown

And a big thank you @usrflo and @mikecarper for the work.

@m0urs

m0urs commented Sep 10, 2026

Copy link
Copy Markdown

As far as I can see it I think that @usrflo it's forwarding of DM is further tested and thought over.

I do see a benefit for flood's also, but with limited resends. If I read @mikecarper his comment it's limited to the first hops. That could be a solution, but maybe also the situation of a busy "air" (lot of traffic) and not a single resend. I see flood's failing on a few hops and that's not nice. I have also a few "high", busy and "strong" repeaters in the neighbourhood, maybe not the best situation.

Making meshcore more reliable is the most important.

I fully agree. I am using @usrflo modification now for several weeks and I can confirm that it is really working. I have an almost 100% success rate with DMs now between two nodes via 4 hops which was never possible with the original firmware.

However I am also experiencing similar issues with flood traffic as @Alain2019 does. Often my channel messages will be sent out by my repeater in the garden but will never spread further. I need to send it one or two times again until it really gets out into the mesh. Unfortuately you will not see that a message is not spread until you double check with an observer.

So maybe having some kind of resend for flood traffic would solve this issue too.

Magalex2x14 added a commit to Magalex2x14/MeshCore that referenced this pull request Sep 11, 2026
Merge of meshcore-dev/MeshCore PR meshcore-dev#2670 (usrflo/MeshCore:feature/repeated-sending-2)
into essentials, applied as a single squashed commit.

Adds a retransmission mechanism for direct-routed packets: after each DIRECT
TX, the originator listens for the downstream relay's forwarding echo and
cancels its own pending resend as soon as it is heard (Packet::isRetryMatch,
Mesh::onRecvPacket). Resends use a non-invasive channel-busy check
(isResendChannelActive) instead of hardware CAD so RX stays open to catch
that echo. Configurable via CLI `set/get max.resend` (0-3, default 2).

Also includes the noise-floor median estimator (replaces the old one-way
mean ratchet that could get stuck at -120dBi) and an RX-desync watchdog
that detects and recovers a radio chip stuck out of RX.

Since essentials and the PR branch had diverged history (both independently
merged upstream/dev), this was integrated as a targeted patch of the PR's
net diff rather than a raw branch merge, with manual conflict resolution
against essentials' existing CAD auto-calibration and NodePrefs layout.
Core packet/dispatcher logic verified to compile against the native test
mocks (test/mocks) and a standalone SimpleMeshTables smoke test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1L5yLTdHL8EkmURkLrN4J
syssi added a commit to syssi/MeshCore that referenced this pull request Sep 25, 2026
After forwarding a DIRECT packet, a repeater schedules up to
max_resend_attempts re-transmissions (default 2, `set max.resend
<0-3>`). Pending resends are cancelled as soon as the downstream hop
is overheard forwarding the same packet (echo cancellation). ACKs get
a dedicated deduplication table, and the RX loop drains all pending
packets per loop() call. Also replaces the one-way noise-floor ratchet
with a median estimator and adds an RX-desync watchdog.

Squashed from upstream PR meshcore-dev#2670 (open, by
usrflo):
- d9f68e9 fix(radio): replace one-way noise-floor ratchet with median estimator
- efc011f docs: add README-bugfix.md for noise-floor median estimator
- f879a9f Removed bugfix documentation for merge request
- f4ebf59 feat: Implement resend mechanism for direct packets
- 3b25ef9 feat: Enhance resend logic with airtime-proportional ceiling and updated fixed latency
- 8acdb11 fix(radio): replace one-way noise-floor ratchet with median estimator
- b87f880 Commented the fixed RSSI margin for DIRECT resend channel activity detection
- b2177fe feat: Add resend ratio tracking and reporting to packet statistics
- b86fcbf feat: Add check for disabled resend attempts in formatResendRatioReply
- 0aeece6 Enhanced noise floor calibration by adding rate-limiting for RSSI samples
- fa6557f feat: implement held block count for noise floor calibration
- 5017924 RX-desync watchdog: verify chip RX mode and self-heal a wedged receiver

Conflict resolution against this branch: kept the CAD auto-calibration
state/defines and the radio CLI settings (dutycycle, af, int.thresh,
cad, agc.reset.interval, multi.acks) that upstream dev has since moved
to CommonRadioPrefs; did not pull in unrelated upstream dev changes
(companion tz_offset, WiFi prefs, getAGCResetInterval).
m0urs pushed a commit to m0urs/MeshCore that referenced this pull request Oct 8, 2026
…pin)

The MeshCore app has no setting for the airtime factor / duty cycle or for
max.resend, so existing companions can now be changed via the rescue CLI.
max.resend needs the PR meshcore-dev#2670 port (merged before mu/mups17-own in mups17).
@m0urs

m0urs commented Oct 20, 2026

Copy link
Copy Markdown

Hi @usrflo, thanks again for this PR – it is still the main reason we run a custom firmware.

Field experience (update to my earlier comment)

We have been running a partial port of this PR on top of IoTThinks PowerSaving-v17 (MeshCore v1.17.1) since <a few weeks now on 4 repeaters and two companions. The port contains the resend logic (Dispatcher/Mesh/Packet, max.resend, resend statistics, non-invasive resend channel check). It leaves out the noise-floor median estimator and the RX-desync watchdog, because both conflict with RX duty-cycle power saving (the radio is legitimately not in continuous RX there).

DM delivery / ACK rate over 4-5 hops: almost 100%
get max.resend (resend ratio) on a busy repeater: about 50%
max.resend setting used: 3

v1.18.0

v1.18.0 has been released. The current head of this PR (33e17bb) no longer applies cleanly on main (81dbae5, v1.18.0): a squashed cherry-pick conflicts in

examples/simple_sensor/SensorMesh.h
src/Mesh.cpp
src/helpers/StaticPoolPacketManager.cpp

src/Mesh.cpp overlaps with #3573 (direct-mode ACK forwarding now uses the retransmit delay), in the same next-hop/ACK block this PR changes.

Would you be willing to rebase the PR onto current dev? It would also help forks like ours if the resend feature and the noise-floor / RX-desync parts could be reviewed separately (the noise-floor part overlaps with #2933).

Happy to run test builds and report numbers from our network.

m0urs added a commit to m0urs/MeshCore that referenced this pull request Oct 20, 2026
m0urs pushed a commit to m0urs/MeshCore that referenced this pull request Oct 30, 2026
The existing "channel busy" counter counts CAD scans that detected
activity, so one packet waiting for a free channel is counted once per
retry (up to ~17 times within the 4 s limit). Add two per-packet counters
in Dispatcher::checkSend(): "blocked" (busy episodes, i.e. send attempts
that had to wait) and "forced" (transmissions started after the max busy
duration). The repeater appends them to the reply of "get cad" and resets
them in clearStats(). Dispatcher::resetStats() is left untouched on
purpose: PR meshcore-dev#2670 edits it, and the merge into mups18 would conflict.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeddNTnt6bXGJhppJ2tAUD
m0urs pushed a commit to m0urs/MeshCore that referenced this pull request Oct 30, 2026
…v#2670 interplay)

Dispatcher::checkSend() returned early when nothing was due, without
clearing cad_busy_start. PR meshcore-dev#2670 can remove a queued resend from the
queue while it waits for a free channel (downstream forward overheard,
Mesh.cpp), so the busy episode was left open with an old timestamp. The
next packet that met a busy channel then looked as if the 4 s limit was
long exceeded and was transmitted at once (and ERR_EVENT_CAD_TIMEOUT was
set). Clear cad_busy_start in the early-return path.

In the official v1.18.0 a queued packet leaves the queue only by being
sent, so this does not occur there. The budget-wait early return is left
unchanged on purpose (same as official).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeddNTnt6bXGJhppJ2tAUD
syssi added a commit to syssi/MeshCore that referenced this pull request Nov 8, 2026
Opt-in packet filter for the repeater ('filter ...' CLI) and a receive
filter for the companion firmware. Both are off by default and store
their settings in their own files (/filter_prefs, /cfilter.json).

Repeater: drop/limit forwarded floods by packet type, hop count, rate,
blocked channels, malformed group text, message age, per-origin advert
window, path-prefix block and sender/text rules (block, throttle, dose),
with dry-run mode and drop statistics ('filter stats').
Companion: hide or drop malformed, too old, path-blocked or rule-matched
group text before it reaches the app; personal rules only hide, the
message is still repeated.

Squashed from Dutch-MeshCore/MeshCore branch dmc-dev at 7df3767
(by Elektr0Vodka, efiten, Marco), filter files only:
- 6b77fbf Add Packet Filter
- a00f970 Add header-only filter drop statistics
- c763532 Add filter stats commands
- 1adee0b Correct the packet type range in the filter error message
- 0158b48 Add probabilistic soft cutoff to repeater packet filter
- 5a2e41a filter: match stats help order to the blocked summary
- d26421b filter: per-origin advert window, path-prefix block, dry-run, saved airtime
- 18341f1 filter: sender and text rules for group text (block, throttle, dose)
- 89dbb2b filter: configurable message age limit for group text (meshcore-dev#8)
- 7656338 refactor(filter): share malformed scan and list editing as headers
- e8b8529 feat(companion): add DMC receive filter
- bff24f5 fix(filter): always build the rate limiters from the active prefs
- 47d3d2d fix(filter): `filter age` also drops messages stamped in the future

Taken over unchanged from dmc-dev 7df3767:
- examples/simple_repeater/{Filter.cpp,Filter.h,FilterStats.h,Limiter.h,
  AdvertLimiter.h,PathBlock.h,SenderRules.h,MessageAge.h,MessageContent.h}
- examples/companion_radio/{CompanionFilter.cpp,CompanionFilter.h,
  CompanionFilterHook.cpp}

Hooks ported by hand (only the filter lines of the dmc-dev MyMesh diff):
- simple_repeater/MyMesh.{h,cpp}: #include, Filter member + ctor init,
  allowPacketForward(), begin() setRadio/load, clearStats(), 'filter' CLI
- companion_radio/MyMesh.{h,cpp}: #include, filter members/methods,
  filterRecvFloodPacket(), onChannelMessageRecv(), begin() load,
  handleCommand()

Adaptation to PR meshcore-dev#2670: Packet::calculatePacketHash() returns a pointer
to a cached hash there, so the two calls in CompanionFilterHook.cpp copy
it with memcpy().

Not taken over: docs, dmc-docs, unit tests (test/test_filter*,
test/test_companion_filter), branding, CI, prebuilts and every other
dmc-dev change (duty-cycle limits, SNMP/OTA, meshcomod import, ...).

To update later: diff the file list above between 7df3767 and the new
dmc-dev head and apply it with 'git apply -3'.
syssi added a commit to syssi/MeshCore that referenced this pull request Nov 8, 2026
Opt-in, off by default. When the repeater's own TX duty cycle exceeds a
threshold, regions are gated from the outermost layer inward: wildcard
'*' first, then the broadest named regions, always keeping the innermost
cluster and the home region open. Once the duty cycle drops below
(threshold - hysteresis) the regions are re-enabled inside-out, with a
random jitter per step to desync recovery across repeaters. The gate is
runtime-only (rt_flags) and never persisted.

CLI: set dc.gate on|off, set dc.gate.thresh <1-100> (default 35),
set dc.gate.hyst <0-50> (default 10), get dc.gate[.thresh|.hyst],
get dc.gate.status (live duty % and gate level).
Prefs are stored via the ConfigSerializer RepeatPrefs group
(dc_gate, dc_gate_duty, dc_gate_hyst), not in the legacy byte blob.

Dispatcher::getTxDutyCyclePercent() is new in this tree (v1.18.0 has
no such method) and is only used by the gate. It reports the true TX
duty cycle: TX airtime / wall-clock over a rolling 60 s window
(src/helpers/TxDutyWindow.h), fed from the single send-complete path
in Dispatcher::loop(), which PR meshcore-dev#2670 resends also pass through. In
Dutch-MeshCore 07e7bbb replaced an earlier budget-headroom metric;
that intermediate version never existed in this tree. The tx_budget_ms
limiter and TX pacing are untouched.

Also included: code-0 TRANSPORT_FLOODs obey the duty-cycle gate on '*'
(e3d029b). This builds on the upstream fix bd882a2 ("scope 0 rule
fixes", added in the previous commit): findMatch() checks
the wildcard's effectiveFlags() (config + runtime gate) instead of its
config flags only, and the code-0 hop-limit check is factored into
RoutingPolicy isUnscopedFlood(). The conflict with bd882a2 in
RegionMap.cpp / RoutingPolicy.h was resolved to the Dutch version.

Squashed from Dutch-MeshCore/MeshCore branch dmc-dev at 7df3767 (by
Elektr0Vodka; 0a65f3c is their port of upstream PR
meshcore-dev#2960, issue meshcore-dev#2747, co-authored by fizzyfuzzle
and Kasper Hägele):
- 0a65f3c Add hierarchical region gating via TX duty cycle (simple_repeater)
- 07e7bbb fix(dispatcher): report true TX duty cycle, not budget headroom
- b12c3fa fix(cli): accept on/off for set dc.gate; clarify dc.gate threshold docs
- 314075e feat(dc.gate): default threshold 35 for true TX duty; rewrite docs
- e3d029b fix(region): code-0 floods obey denyf * and flood.max.unscoped

Apart from the bd882a2 overlap, the code applied without conflicts.
Dropped: all documentation changes
(docs/cli_commands.md, docs/DMC_DEVSYNC.md, dmc-docs/*).
Kept: native unit tests (test/test_region_gating, test_routing_policy
additions), the [env:native_region_gating] env and the test mock
extensions they need.
syssi added a commit to syssi/MeshCore that referenced this pull request Nov 8, 2026
Dispatcher::checkSend() returned early when nothing was due, without
clearing cad_busy_start. PR meshcore-dev#2670 can remove a queued resend from the
queue while it waits for a free channel (downstream forward overheard,
Mesh.cpp), so the busy episode was left open with an old timestamp. The
next packet that met a busy channel then looked as if the 4 s limit was
long exceeded and was transmitted at once, and ERR_EVENT_CAD_TIMEOUT
was set although nothing hung. Clear cad_busy_start in the early-return
path. The budget-wait early return is left unchanged on purpose: the
packet is still queued there.

In the official v1.18.0 a queued packet leaves the queue only by being
sent, so this only occurs together with PR meshcore-dev#2670. Should be reported
upstream on meshcore-dev#2670; drop this commit once the PR
contains the fix.

Cherry-picked from m0urs/MeshCore branch mu/mups18-own (by m0urs),
src/Dispatcher.cpp only:
- 5150077 mups: reset the busy-wait timer when no packet is due (PR meshcore-dev#2670 interplay)

Not taken over: the MUPS18.md entry; the 'mups:' prefix was dropped
from the code comment.
@m0urs

m0urs commented Nov 15, 2026

Copy link
Copy Markdown

@usrflo @Kernel-Error

Two TRACE issues in the resend logic of this PR

While porting this PR to a downstream firmware (MeshCore v1.18.0 + IoTThinks PowerSaving), Sebastian van de Meer (GitHub: Kernel-Error) found two TRACE-related issues, and I reviewed them. The relevant code (Mesh.cpp, Packet.cpp, Dispatcher::resendPacket()) is the same in this PR as in my port.

  1. Queued resends of TRACE forwards are never cancelled

In Mesh::onRecvPacket() the TRACE branch at the top always returns (ACTION_RETRANSMIT_DELAYED or ACTION_RELEASE). The generic “downstream forward overheard → cancel queued retransmit” block further down is documented to handle TRACE via Packet::isRetryMatch(), but it is never reached for TRACE. A relay in the middle of a TRACE path has appended its SNR, so the path hash count is > 0. resendPacket() therefore queues retries, and nothing cancels them. Each intermediate hop re-sends every TRACE up to getMaxResendAttempts() times, even though the next hop has already forwarded it. The next hop drops those copies as already seen, so this is wasted airtime.

Suggested fix (m0urs#25): inside the TRACE branch, before the normal processing, scan the outbound queue for a queued TRACE resend (sending_attempts > 0, not the terminal leg) that matches pkt->isRetryMatch(q), then remove and free it. After that, process the received packet as usual, because a TRACE may legitimately revisit the node on the return path.

  1. The last-hop TRACE forward is treated as a resend

To suppress retries on the last forwarding hop, the PR sets pkt->sending_attempts = getMaxResendAttempts(). As a result, the first (and only) send already has sending_attempts > 0. Dispatcher::checkSend() then uses the resend gate (isResendChannelActive()) instead of the normal first-send channel check, a busy result uses the resend re-check cadence, and the send is counted in n_resent_direct. The number of transmissions is correct (one), but the gate and the statistics are wrong. A false “busy” from the resend gate can also delay the packet up to the CAD fail max duration.

Suggested fix (m0urs#26): a runtime-only bool no_retry on Packet. It is set instead of pre-setting the counter, cleared in the constructor and in StaticPoolPacketManager::free(), and checked first in resendPacket().

Both changes build for Heltec V3 and RAK4631 on my side. Kernel-Error reports field measurements in the linked PR #26, which I have not verified myself.

usrflo and others added 5 commits November 17, 2026 21:01
…ation

After forwarding a DIRECT packet, a repeater schedules up to
max_resend_attempts re-transmissions (default 2, 0-3 via 'set max.resend').
Each pending retransmit is cancelled immediately when the repeater
overhears the downstream relay forwarding the same packet (echo-detection
via cached packet hash in the outbound queue). The final relay hop of
ACK-producing packets (TXT_MSG) gets exactly one ACK-cancellable resend.

- resend window W = C0 + K*airtime + margin + (attempt-1)*jitter, under an
  airtime-proportional ceiling so it never clips long packets / high SF
- resends use a non-invasive channel gate (isResendChannelActive) so the
  radio stays in RX and can overhear the cancelling forward
- deterministic per-hop busy-recheck buckets (pathHashCount % 3) keep
  deferred resenders from colliding on the same loop tick
- pool-shedding: resends are skipped under low free-pool pressure
- ACK deduplication via a dedicated table (_acks[]) to keep dedup cheap
- 'get max.resend' reports the resend ratio of direct TXs

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ast-hop TRACE

Two TRACE issues in the resend logic, found by Sebastian van de Meer
(Kernel-Error) and reviewed/measured by m0urs on a live repeater
(m0urs#25, m0urs#26):

1. Queued resends of TRACE forwards were never cancelled: the TRACE branch
   in Mesh::onRecvPacket() always returns, so the generic echo-cancel block
   further down was never reached for TRACE. Each intermediate hop re-sent
   every TRACE up to max_resend_attempts times even though the next hop had
   already forwarded it (~40 redundant transmissions per 30 min on a busy
   repeater). The TRACE branch now scans the outbound queue for a matching
   queued TRACE resend (isRetryMatch) and removes it, for well-formed
   non-terminal legs, before normal TRACE processing.

2. The last-hop TRACE forward pre-set sending_attempts = max to suppress
   retries, which made the first (and only) send look like a resend: it went
   through the resend LBT gate instead of the normal channel check and was
   counted in the resend statistics. Replaced by a runtime-only no_retry
   flag on Packet (reset in the ctor and in the pool free()), checked first
   in resendPacket(). The packet is sent once like any other first send and
   never resent.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Drop claims that no longer match the code (RX-FIFO draining, relocated
noise-floor calibration, RESEND_BACKOFF_STRETCH_MS, stale commit hashes),
document the airtime-proportional cancel-window model, the TRACE cancel
scan and the no_retry flag, the ConfigSerializer persistence, and the
resend-ratio output of 'get max.resend'.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The max.resend command existed only on CommonCLI nodes (repeater, chat,
room server). Companion radios run their own command dispatcher in
MyMesh::handleCommand() (reachable over USB/BLE via CMD_RUN_CLI_COMMAND),
so app users could set the value only through the raw CMD_SET_OTHER_PARAMS
byte but not read it back or discover it through the CLI.

Add the command to the companion dispatcher with the identical command
syntax and reply format as CommonCLI, wired to the companion's own
NodePrefs (max_resend_attempts, 0-3, persisted via savePrefs()). The
get-reply appends the same resend-ratio suffix (', resends N/M (P%)')
from the Dispatcher counters; a plain '> 0' is kept when the feature is
disabled, mirroring the repeater behaviour.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@usrflo
usrflo force-pushed the feature/repeated-sending-2 branch from 33e17bb to 9ef20a0 Compare November 18, 2026 12:52
@usrflo

usrflo commented Nov 18, 2026

Copy link
Copy Markdown
Author

@m0urs @Kernel-Error thanks a lot — both issues are real, and both are fixed now, following your suggested fixes (m0urs#25, m0urs#26). The branch is also rebased onto current dev and restructured:

Rebase / separation (from your October note). The branch is now dev@ca580461 + 5 commits, rebuilt without the merge history. The noise-floor median estimator and the RX-desync watchdog are completely removed from this PR, as you suggested — isResendChannelActive() simply calls the existing getNoiseFloor() interface, so the resend feature now contains zero radio-driver changes. The estimator lives in #2933 alone; the two PRs are complementary and can be merged independently (with #2933, the 12 dB resend margin works against a floor that doesn't drift to −120 dBm).

1. Queued TRACE resends never cancelled — fixed with your approach: the TRACE branch in Mesh::onRecvPacket() now scans the outbound queue itself (payload/SNR-prefix match via Packet::isRetryMatch(), only well-formed non-terminal legs) and drops a queued TRACE resend when a later hop's forward is overheard, before normal TRACE processing.

2. Last-hop TRACE treated as a resend — also fixed as suggested: a runtime-only no_retry flag on Packet (cleared in the constructor and in StaticPoolPacketManager::free(), checked first in resendPacket()) replaces the sending_attempts = max pre-set. The last-hop forward now goes through the normal first-send gate and no longer inflates n_resent_direct.

Sim A/B for both fixes (mcsim, trace_retry topology — 3 marginal hops at ~87 % per-packet loss, 8 TRACEs per run, 3 seeds):

TX total Delivered
without TRACE fixes 188 23/24
with TRACE fixes 102 (−46 %) 24/24

Per-seed TX: 64→35, 64→32, 60→35. The intermediate hops went from 3× TX per TRACE (every queued resend fired, none ever cancelled) to a handful of resends total; the terminal hop sends exactly once. This matches your live measurements (~40 redundant retransmissions per 30 min saved, no delivery change).

Hardware validation (6-node bench). We also validated the branch on real hardware: two companions + two repeaters, forced 3-hop DIRECT route A→R1→R2→B (out-paths pinned on both ends, so DM and ACK traverse the full chain), plus a controllable LoRa interferer and a passive RF observer logging every over-the-air packet with ms device timestamps. SF8 / 62.5 kHz / CR 4:5, 22 dBm, all nodes in mutual RX range (worst case for echo/dedup logic). Evidence = per-hop observer log cross-checked against node stats.

  • Clean delivery — exactly one TX per hop, both directions. 160-char DM (the longest single-packet DM) with its returning ACK: 3 TXT hops + 3 ACK hops, each transmitted exactly once, 0 resends, 0 duplicates; repeater stats match 1:1 (2 originals + 2 overheard echoes correctly deduped per repeater).
  • Induced mid-chain loss → one resend, after the window, chain recovers. Burst-jamming a repeater's forward mid-TX: the next hop misses it, and the repeater retransmits the identical packet after ~5.0 s (device-ms measured) — consistent with the configured window W = C0 + K·airtime (1500 + 2×1160 + 40 ≈ 3.9 s) plus the observed ~0.4–0.7 s forward-scheduling latency. The origin does not resend: it overheard the (killed) forward and its pending resend was cancelled — echo-cancellation working live.
  • Interference before the forward → deferral, not collision. A burst already on air when the repeater schedules its forward makes it wait for a clear channel (no collision, no resend, delivery intact).
  • Resend gate over a 48-min run — no drift. 32 periodic DMs (60/160-char alternating), every 3rd burst-jammed: 32/32 delivered incl. all ACK return routes; final accounting exact (sent = forwards + resends on both repeaters). The window is deterministic on hardware: 4.973/4.974/4.974 s across three loss events for the 160-char case, 3.499 s for the 60-char case — the 1.48 s delta matches the designed 2×airtime scaling. Resend ratio flat at 3–6 % over the whole run; no conservative drift at this timescale without Fix/noise floor ratchet, adapt noise to the real floor with recovery in both directions #2933.
  • TRACE last-hop no_retry — live proof. Clean TRACE forwards exactly once per hop with the per-hop SNR trail visible in the over-the-air packet; the destination's reception is proven by its reply (the origin's trace-data push carries the destination's measured SNR). Burst-jamming the terminal delivery: the destination missed it (reply-SNR absent), and the terminal repeater sent that final hop exactly once, with no retry ever — while the identical burst pattern at the middle hop made the first repeater retry its forward. So no_retry demonstrably prevents the un-cancellable retry storm into the destination.

Both fixes build for Heltec V3, RAK4631 and the companion (BLE); the native test suite passes 50/50. The companion CLI now also supports set/get max.resend (same syntax and reply format as the repeaters, via the app's CMD_RUN_CLI_COMMAND passthrough), so get max.resend ratios can be collected on companions too — if you can run a test build of this branch in your network, before/after numbers for the no_retry change would be very welcome. Raw per-hop logs (JSONL, ms device timestamps) available on request.

@Kernel-Error: the well-formedness checks from your PR#25 (plain SNR count, whole route entries, no more SNRs than hops, terminal-leg exclusion) are carried over verbatim — please take a look whether the port into this branch matches your intent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants