feat(compiler): incremental compile cache & warm hot-reload daemon (REFLEX_COMPILE_CACHE)#6688
feat(compiler): incremental compile cache & warm hot-reload daemon (REFLEX_COMPILE_CACHE)#6688FarhanAliRaza wants to merge 18 commits into
Conversation
Add an experimental, flag-gated incremental frontend compile cache that recompiles only the pages whose source actually changed and reuses the rest. Two layers, both off by default and enabled by REFLEX_COMPILE_CACHE: - In-process per-page cache (page_cache.py): a Salsa-style dependency graph records the exact set of source files each page reads, so editing one file invalidates only the pages that depend on it. Pages are keyed by a small genuinely-global epoch (Reflex version + rxconfig + lockfile) plus the content hashes of their dependency set. Speeds up repeat compiles within a single process. - On-disk manifest (disk_cache.py): persists each page's serializable contribution and dependency hashes to .web/reflex_compile_cache.json so a fresh process — notably a `reflex run` hot-reload worker, which respawns on every edit — recompiles only changed pages and reuses the rest. Falls back to a full compile on any unsafe condition. REFLEX_COMPILE_CACHE_VERIFY runs a full compile alongside the cached one and asserts byte-identical output, falling back on mismatch — the backstop for gaps a static dependency graph cannot see (runtime importlib imports, data read at module-import time). Supporting changes required for safe page reuse: deterministic compile-time ref-name generation, and own-before-mutate page metadata injection.
Merging this PR will not alter performance
Comparing 
Footnotes
|
Greptile SummaryThis PR adds an experimental incremental frontend compile cache. The main changes are:
Confidence Score: 4/5This is close, but one cache invalidation path should be fixed before merging.
reflex/utils/prerequisites.py Important Files Changed
Reviews (16): Last reviewed commit: "test(compiler): scope contexts snapshot ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a453ccbd69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add a persistent compile daemon (REFLEX_COMPILE_CACHE) that imports the world once and forks a throwaway child per source change, instead of the reloader respawning a worker that cold-imports on every edit. The child re-imports first-party code fresh and runs the incremental rebuild, so correctness matches a respawn while the cold import is paid once. Supporting changes that make the daemon safe and complete: - write_file now writes atomically (temp + os.replace) so a reader (vite, a concurrent compile) never sees a half-written file, even when a forked child is killed mid-compile. - _run_dev launches the daemon and sets REFLEX_SKIP_COMPILE on the backend so it only evaluates pages to register state. - The daemon watches what the compiler reads (incl. sibling-dir markdown from the manifest); uvicorn reload_includes also covers *.md/*.mdx, so markdown edits finally trigger a reload. - Drop the per-rebuild console.info now that progress is shown inline.
multi_docs built every component's prop tables at module import, so the whole library reference was reconstructed on every import of the docs tree -- re-run on every dev hot-reload reimport, cold start, and backend respawn. Move the build into the page render closures (matching the non-library doc path) so a page builds its prop tables only when it is actually compiled. Docs cold import 9.7s -> 1.9s; hot-reload reimport 8.3s -> 0.6s.
- disk_cache: stop re-evaluating stateful HIT pages during an incremental rebuild. The compiling process never serves (the daemon, the initial compile, and CLI compiles all exit; the serving backend re-evaluates the marked stateful pages itself), so re-running their render pipeline was pure waste. The stateful-pages marker stays complete -- hits recorded from the manifest, misses from the fresh compile. - compile_daemon: poll faster (0.25s -> 0.05s) but cheaply -- stat the known file set each tick and rglob only every 1s for added/removed files, cutting detection latency without burning idle CPU. - compile_daemon: log per-edit timing (reset / reimport / compile).
|
@greptile please rereview |
Profiling the hot-reload compile on the docs app (418 pages) showed the disk-cache rebuild spent ~1.1s of its ~1.6s on manifest I/O: the manifest stored every page's rendered output_code (and output_path/frontend_imports), ballooning it to 46MB, yet those fields are never read back — only dep_hashes, app_wrap_keys, is_stateful, and the merged all_imports are consumed. Storing just the bookkeeping that is read drops the manifest to 14MB. The in-process page cache (_PAGE_STORE / validate_page / store_page) is never reached by the warm daemon (its fork child cleared it and the disk path returns first), and forcing it measured ~55x slower than the disk path because it re-runs the whole app-level pipeline (memo render, stylesheet, plugins) every edit. Remove it, the verify mode that only guarded it (REFLEX_COMPILE_CACHE_VERIFY), and the dead page_source_fingerprint. Manifest schema bumped 2->3 so stale fat manifests are ignored.
Remove page_module_files and file_hashes, now-dead after the compile-cache manifest rework, along with their tests.
Two correctness fixes for the incremental compile cache, plus a pass to trim verbose comments and docstrings across the cache modules. - global_epoch now fingerprints the app-level config files: the app entrypoint module plus the config-only modules it imports (theme, app-wraps, stylesheets), found by walking the import graph with page modules as barriers so per-page incrementality is preserved. An edit to app-wide config previously left every page a hit, so the reused on-disk app root / contexts / theme stayed stale. - The incremental manifest refresh now persists the complete install import set (page imports merged with the memo-component imports the rebuild generated, and threading root through dependency discovery), so a later all-hit compile installs every package the reused memo files need instead of dropping newly-introduced memo packages.
test_reset_model_metadata_allows_table_redefinition builds an
rx.Model(table=True), which needs the SQLModel stack. The "without db
dependencies" unit-test job uninstalls it, so the first model definition
raised TypeError before the test could exercise the reset. Guard with
pytest.importorskip("sqlmodel"), matching the repo's other db-dependent
tests, so it skips there instead of failing.
Add the news fragments the changelog check requires for the packaged source touched by this PR: a feature entry for the reflex package's REFLEX_COMPILE_CACHE flag and a misc entry for the reflex-base recorder hook / reproducible ref names that support it.
Patch __import__ and importlib.import_module while a page recorder is active so modules imported during page eval are recorded as source dependencies. Resolves names (including relative imports), skips stdlib and out-of-project files, and caches module-file lookups. Lets the incremental compile cache invalidate pages that depend on first-party modules pulled in at import time.
Replace the os.path string-comparison helpers used for runtime import tracking with pathlib.Path operations, dropping the redundant cached root-string globals (_recorder_root_str, _recorder_raw_root_str) and the sentinel-based module-file cache. Make the read-set target an explicit argument instead of falling back to the recorder context.
The compile-cache read tracker patches builtins.__import__ to record a page's runtime imports. While recording, _recordable_module_file resolves a module file via Path(...).absolute(), which on CPython 3.12 lazily runs `import ntpath` to compare path flavours. That import re-entered the patched hook -> resolved a path again -> imported again, recursing until the stack overflowed (32 unit tests failed only on ubuntu 3.12). Suspend read/import tracking (_suspend_tracking) while the tracker's own path/module bookkeeping runs, so imports and reads it triggers are neither recorded nor able to re-enter. Add a version-independent regression test that mimics the 3.12 lazy import.
|
@codex pelase review |
Config the app module pulls in dynamically (importlib calls, files read at import time) was invisible to app_dependency_files, which only walked the static import graph from the entrypoint. Editing such config could leave reused app-wide output stale. Record files read/imported while the app module loads (record_app_import, wired into prerequisites.get_app under REFLEX_COMPILE_CACHE) and fold that dynamic set into app_dependency_files, subtracting per-page static closures so ordinary page edits still invalidate per-page. Extract the shared graph walk into _walk_import_closure.
FarhanAliRaza
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 451be1e96a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| memo_defs = list(page_ctx.memo_contributions.values()) | ||
| memo_files, memo_imports = compiler.compile_memo_components(memo_defs) |
There was a problem hiding this comment.
When a changed page uses @rx.memo or shares a mirrored memo module with a hit page, page_ctx.memo_contributions is not a complete set of definitions for that output file because it excludes global MEMOS and hit-page auto memos. Since compile_memo_components() emits one whole file per mirrored source module, writing only this subset can leave edited @rx.memo code stale or overwrite the module without exports still imported by reused pages.
Useful? React with 👍 / 👎.
Four reviewer-confirmed ways the incremental path could serve stale .web output: - Path.open() calls io.open directly, bypassing the builtins.open patch, so data files read through it were never recorded as page dependencies. Patch Path.open itself. - The contexts file was never re-emitted, keeping old state defaults and client-storage config after a state-module edit. Re-emit it on every incremental rebuild (write_file already skips byte-identical writes). - assets/ was never copied (excluded from dependency tracking and the epoch), so an assets-only edit produced an all-hit rebuild with a stale .web/public. Run the same mtime-incremental copy as the full compile. - Memo output files are grouped one file per source module, but a miss rewrote them from only its own auto-memo contributions, dropping user @rx.memo exports and hit-sibling exports still imported by reused pages. Recompile memo-contributing same-module siblings together (tracked by a new has_memos manifest flag, schema 3 -> 4) and include user memos that share a rewritten file or whose module changed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21e7d3ea9e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| #: Watchdog: kill a compile child that runs longer than this (a hung/deadlocked | ||
| #: child must never wedge the daemon). Generous enough for a real full compile. | ||
| _COMPILE_TIMEOUT = 300.0 | ||
| #: Source suffixes edited under the app roots that should trigger a recompile. |
There was a problem hiding this comment.
When a page reads an in-project data file with a suffix that record_reads() records (such as .json, .yaml, .toml, .txt, .csv, .html, or .rst), editing that file does not wake the compile daemon: this watched suffix list only includes .py, .md, and .mdx, and _external_dependency_files() filters out deps already under the reload roots. The manifest would make the page stale if a compile ran, but no compile is triggered, leaving the frontend stale until another watched file changes.
Useful? React with 👍 / 👎.
The incremental rebuild no longer re-emits the contexts file every time. It is rewritten only when a stateful page missed and that page's own state config actually changed (fingerprinted against the manifest); otherwise the on-disk file is reused untouched. When a rewrite is needed, the stateful hit pages are evaluated first so the state registry — and the frontend dispatch map compiled from it — stays complete. Supporting changes: - Reset the daemon's state registry surgically: states from modules that survive the purge (framework/installed/workspace packages) are re-registered in original order, while purged-module and reflex.istate.dynamic states are dropped so re-created local states get deterministic fresh-process names. - Patch react-router's served HMR runtime so an edit to a not-currently-open route no longer throws and poisons HMR until a full reload. - Store per-input epoch digests instead of one combined hash so a global-input mismatch can name the exact file that changed, and surface fallback reasons and detected file changes at info level. - Write the vite config atomically and regenerate it on the incremental path.
The hot paths of a warm rebuild did far more work than needed: - Collapse duplicate ImportVars when serializing the manifest and merge the miss pages' import sets once, instead of re-merging the ~100k-entry app-wide set per page. A full docs-app compile accumulates ~107k import entries of which only ~6k are unique. - Key the module-file cache by the raw __file__ value and check it before any Path construction, so every import statement under an active read recorder skips the resolve. Cleared when the recorder root changes. - Compare path strings in _under_roots instead of building a Path per ancestor per module, which dominated the daemon's reset phase on every hot reload.
The incremental-cache tests fingerprint state config via _contexts_snapshot, which walks the whole root state tree — picking up unrelated (and sometimes broken) state classes registered by other test modules. Stub it with a snapshot limited to the states this file defines, and stub externals before write_manifest so the fingerprint it records uses the same scoped snapshot.
| app = ( | ||
| __import__(module, fromlist=(constants.CompileVars.APP,)) | ||
| if not config.app_module | ||
| else config.app_module | ||
| ) |
There was a problem hiding this comment.
config.app_module is already set, this branch returns the existing module object inside record_app_import() without re-importing the app or replaying the dynamic config imports. That records an empty app import read set for the project root. With REFLEX_COMPILE_CACHE enabled, editing a dynamically loaded theme, app wrap, toaster, or provider module can leave the global cache inputs unchanged, so the incremental rebuild can reuse stale app-root output instead of falling back to a full compile. This path needs to either fingerprint dependencies from the preloaded module or force a fallback when those reads cannot be captured.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dbfacec09
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if root in rf.parents: | ||
| file_to_mod[str(rf)] = name |
There was a problem hiding this comment.
When an app uses REFLEX_HOT_RELOAD_INCLUDE_PATHS for a sibling local package, the daemon watches that root, but this dependency graph keeps only modules whose files are under Path.cwd(). A page importing a component/helper from the sibling package therefore records no hash for that .py file; editing it wakes the daemon, partition_pages sees all stored deps unchanged, and the incremental path reuses stale .web output.
Useful? React with 👍 / 👎.
| _record_module_file(result, target) | ||
| result_id = id(result) | ||
| if (module := sys.modules.get(name)) and id(module) != result_id: | ||
| _record_module_file(module, target) |
There was a problem hiding this comment.
This records only the module returned by a runtime import. If a page dynamically imports a helper that was already loaded by an earlier page or by app import, the helper's own imports are not executed under this recorder and are not added to the page dependency set; editing one of those transitive first-party files can leave that page classified as a cache hit and keep its previously generated output stale.
Useful? React with 👍 / 👎.
1 participant
An experimental, flag-gated frontend compile pipeline that recompiles only the pages whose source actually changed and reuses the rest. Off by default; enabled by
REFLEX_COMPILE_CACHE. The compile path is completely unchanged when the flag is unset.What it does
On-disk incremental cache (
disk_cache.py+page_cache.py)A fresh compile process reuses the previous build already on disk in
.weband recompiles only the pages whose source changed. Each page is invalidated by a precise dependency set rather than a global timestamp:importlibimports and data read at import time),Pages are keyed by the content hashes of that set plus a small global epoch — Reflex version,
rxconfig/lockfiles, and the app entrypoint's config-only modules (theme, app-wraps, stylesheets, head components, plus config the app imports dynamically at load time). A change to a genuinely-global input forces a full recompile; per-file edits invalidate only the pages that depend on them. Any unsafe condition falls back to a full compile, so a cache miss is never a correctness risk.Warm fork-per-compile hot-reload daemon (
compile_daemon.py)reflex rundev imports the app once, then compiles each edit in an isolated child — forking on POSIX so third-party imports stay warm (falling back to a fresh process elsewhere). Combined with the disk cache, a hot reload skips the cold reimport and rebuilds only what changed.Supporting changes required for safe page reuse: deterministic compile-time ref-name generation, own-before-mutate page-metadata injection, and mirroring memo output paths to their Python source modules.
All Submissions:
Type of change
New Feature Submission:
Changes To Core Features: