fix(traefik): stop Let's Encrypt from overriding domains set to Certificate: None - #4963
fix(traefik): stop Let's Encrypt from overriding domains set to Certificate: None#4963onlyilkr wants to merge 15 commits into
Conversation
The store was rewritten in place on both the local and the remote path. The remote command truncated acme.json with `>` before writing and used `;` to chain chmod, so a dropped SSH stream, a failing base64 or a full disk left an empty, world-readable store behind and destroyed every Let's Encrypt certificate on that server. `fs.writeFileSync` had the same truncate-in-place exposure. Both paths now write a sibling temp file, set mode 0600 on it and rename it over the target, which is atomic within a directory. The remote commands are chained with `&&` so a failure stops the sequence, and the temp file is removed on failure.
The helper lived inside the tRPC router, so the shared-host guard, the only thing preventing the deletion of a certificate another domain still needs, had no test coverage. It now lives next to the domain services it uses and is exported, with unit tests for the purge, the guard, the letsencrypt short circuit and the failure path. A purge failure no longer fails the mutation either: the domain row was already updated and the router already regenerated, so an unreachable server made the user see an error for an update that succeeded. Failures are logged and reported as "no reload required".
The startup pass purged ACME certificates without the shared-host guard the mutation path applies, so a host still served by another domain with certificateType "letsencrypt" lost its certificate at boot and Traefik was restarted on top of it. It now reuses hasOtherLetsencryptDomainForHost and only purges genuinely unused hosts. The purge also ran after manageDomain. Once the router carries its `tls` key, routerNeedsTlsFix is false, so a purge that threw was never retried on a later boot. Purging first leaves the router untouched on failure and the next boot retries the whole domain.
…hange The domain update mutation already returned traefikReloadRequired but no caller read it. Traefik only reads acme.json at startup, so without a restart the purge has no effect and can be undone when Traefik next rewrites the store from memory. The dialog now raises a toast when the field is true.
The four reconciliation behaviours were asserted inside one large `it` with a sentence-long title, so a failure said nothing about which one broke. They are now separate cases over a shared beforeEach fixture, plus coverage for the shared-host guard and the purge-before-regenerate ordering.
Raising it as a second toast collapsed it onto the success one, since the Toaster does not set expand, and no other place in the app fires two toasts for a single action. It is now the success toast's description, with a longer duration because the message asks the user to go and restart Traefik rather than just confirming what happened.
dosubot
Bot
added
size:L
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
A purge is a read-modify-write over a file Traefik owns. On a remote server the read is an SSH round trip, so two domains switched off Let's Encrypt at the same time both read the original store and the later write puts back whatever the earlier one removed, leaving a stale certificate that keeps shadowing the Origin CA one. Purges for a server now run one at a time. The chain survives a failed purge so one broken SSH connection cannot block later ones.
The domain query ran outside the per-application error isolation, so a database failure propagated to the shared catch in server.ts. Everything sequenced after it was skipped: the backup cron jobs, the restart notifications, the enterprise cron jobs and the deployment worker, all while the HTTP server was already listening. The pass is best effort and now never rejects.
| result.then( | ||
| () => undefined, | ||
| () => undefined, | ||
| ), |
There was a problem hiding this comment.
If Traefik issues or renews a certificate after this purge reads acme.json but before it renames its modified snapshot over the file, the in-process lock does not coordinate with Traefik's write, causing the newly issued or renewed certificate to be lost from persistent storage.
Knowledge Base Used: Traefik Networking
There was a problem hiding this comment.
Mitigated in d46f6e4 rather than closed, and I want to be straight about the difference.
The store is now re-read and compared just before the swap and the purge starts over if Traefik changed it underneath. That narrows the window to the final round trip. It does not close it: Traefik owns this file, rewrites it in full on issue or renewal, and offers no locking protocol to join.
What is left: if Traefik writes inside that window, the certificate it just issued is lost from acme.json. It stays in Traefik's memory and keeps being served, so nothing breaks at runtime. The loss only surfaces after a restart, when Traefik re-issues it, and the startup reconciliation re-purges the stale one, so the state converges.
I did prototype the version that closes it properly: stop Traefik, purge, start it again. Editing the file with no writer running removes the race outright, and it costs nothing extra because the purge only takes effect across a restart anyway. I am not folding it into this PR. It trades a bounded, self-healing data race for an availability risk, since a failure between stop and start can leave Traefik down, and on the local server that takes the Dokploy UI with it. It also changes what settings.reloadTraefik does, which is beyond what this PR set out to fix.
Happy to open that as a follow-up PR so it can be judged on its own, or to drop the "Dokploy writes to acme.json" approach entirely if that file should stay Traefik's alone. @Siumauricio whichever you prefer.
There was a problem hiding this comment.
The analysis in d46f6e4 is accurate and well-reasoned. The remaining window — Traefik writes between your re-read and mv — has exactly the convergence property you describe:
- Lost cert stays in Traefik memory, keeps being served
- After restart, Traefik re-issues it
- Startup reconciliation purges the stale router config again
That's a self-healing loop, not data corruption. The original concern was about silently destroying a certificate Traefik just issued. After d46f6e4, the only scenario where that still applies is a write inside the final read → mv round trip, and even then the consequence is a re-issue on next restart rather than a user-visible outage.
The stop-Traefik approach would genuinely close it, but you've correctly identified the tradeoff: availability risk on failure between stop and start, and it changes reloadTraefik semantics. Opening it as a standalone PR is the right call — it can be evaluated against a dedicated risk/benefit question rather than bundled into a bug fix.
One minor nit on the current code: the PURGE_ATTEMPTS = 3 retry loop gives up after 3 consecutive Traefik writes and returns [], logging a warning. That's fine for correctness, but the caller in the domain-save path has no way to distinguish "nothing to remove" from "we gave up" — both return an empty slice. If the warn path matters for observability, consider returning a distinct signal (or throwing) so the caller can surface it differently in the UI toast.
Otherwise the implementation looks solid and the comment at lines 127–133 clearly documents the residual risk and why it's acceptable. I'm happy to mark this concern as addressed given the mitigation and the convergence guarantee.
Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.
Traefik owns acme.json and rewrites it in full whenever it issues or renews a certificate, with no locking protocol to join. A snapshot taken before such a write would have dropped the new certificate on rename. The store is now re-read and compared just before the swap, and the purge starts over if it changed. That narrows the window to the final round trip rather than closing it, which is as far as this can go without stopping Traefik to edit its own state. If a purged certificate is reinstated by a write we lost the race to, the startup reconciliation purges it again on the next boot.
1 participant
What is this PR about?
A domain set to
Certificate: Nonekeeps being served its old Let's Encrypt certificate, and Traefik keeps trying to renew it. Once that certificate expires, Cloudflare inFull (Strict)mode returns Error 526 even though a valid Cloudflare Origin CA certificate is installed and the domain looks correctly configured.The cause is not that the
websecureentrypoint carries acertResolverdefault. It is that the router never overrides it. Traefik applies an entrypoint's TLS default only to routers whose TLS is nil (pkg/server/aggregator.go,applyModel):and
certificateType: "none"wrote exactly that nil. Writing an empty TLS block instead leaves the entrypoint untouched, so there is notraefik.ymlmigration and no risk oftls:ending up null, which would disable TLS and turn 526 into 525 across every site.utils/traefik/forward-auth.tsalready handled the identical case this way, so this part is really a consistency fix.Three changes:
1. Routers override the entrypoint default.
certificateType: "none"now writestls: {}. The same widening coverscertificateType: "custom"with an emptycustomCertResolver, which previously matched no branch and fell through to nil as well. The Docker-label path gets the equivalent change; it already emittedtls=truefornone, so Compose domains were never hit by the original bug.2. The stale certificate is removed. Fixing the router stops new issuance but leaves the old certificate in the store, and an exact-hostname Let's Encrypt certificate still wins over a wildcard Origin CA one. When a domain moves off Let's Encrypt its entry is now dropped from
acme.json, guarded so a hostname another domain still serves with Let's Encrypt is left alone. The store is written atomically (temp file,0600, rename) rather than truncated in place, so a failed write cannot destroy the certificates on that server.3. Existing installations are reconciled.
manageDomainruns on domain create and update but not on deploy, so an installation that already has the broken router config on disk would not pick the fix up until every affected domain was re-saved by hand. A one-shot pass at startup regenerates only the router configs that are actually missing their TLS key. It is idempotent, it preserves manual edits becausemanageDomainreplaces only its own router and service keys, and failures are isolated per application so an unreachable remote server cannot block startup.Things worth knowing while reviewing
acme.jsonis read once at startup and served from memory afterwards. Restarting Traefik briefly drops every site on the server, which is too much of a side effect for saving one domain, so the mutation reports it and the success toast tells the user to restart. The startup pass does reload automatically, once per server and only when something was actually removed, because at boot that is far less disruptive. Happy to change either behaviour if you would rather have it the other way.applicationId.certificate.ymlis read at startup and on any reload triggered by a top-level change, just not on its own. Different problem.Checklist
canarybranch.Issues related (if applicable)
closes #4949
Testing
Unit tests cover all three certificate types on both the file-config and the Docker-label path, the ACME store helper including the shared-host guard and the failure path, and the reconciliation pass. 105 tests pass across the affected files, and
packages/serverandapps/dokploytypecheck clean.Verified end to end on a local instance:
Certificate: Noneon a domaintls: {}Let's Encrypttls: { certResolver: letsencrypt }, unchanged from beforeNoneacme.json, the other certificate untouched,Accountpreserved, file still0600, no temp file left behindtlskey deleted by hand, then a restartReconciled TLS config for 1 domain(s) on <app>, key restoredScreenshots (if applicable)
Greptile Summary
This PR changes application-domain TLS generation so Certificate: None explicitly overrides Traefik’s entrypoint resolver, removes obsolete ACME certificates, and reconciles affected existing installations.
Confidence Score: 4/5
The PR is not yet safe to merge because the ACME purge can still overwrite certificate state written by Traefik during the final check-to-rename window.
The new reread detects Traefik changes made before the comparison, but both write paths replace acme.json afterward without coordination; a renewal or issuance in that interval is therefore still lost from persistent storage.
Files Needing Attention: packages/server/src/utils/traefik/acme.ts
Reviews (3): Last reviewed commit: "fix(traefik): re-check acme.json before ..." | Re-trigger Greptile
Context used (3)