perf: large-repo render + load pass (#75)#109
Merged
Conversation
Times each manifest→city apply phase (layout, worker-payload clone, bbox, buildings assembly, media ad-panel registration, street labels, picker raycast) for two workloads: raw scale (80k files, Linux-ish) and image-heavy (PostHog-ish). Profile-first driver for #75 so fixes target the phase that actually dominates. Key caveat baked into the file: the street-label phase is jsdom-inflated (node-canvas rasterizes text ~100x slower than a browser GPU canvas), so its absolute ms is not browser-representative — the machine-independent label cost (geometry/mesh allocation) is small. The trustworthy, machine-independent numbers are layout (off-thread), buildings assembly, clone, bbox, and the picker raycast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the "load PostHog, zoom out + rotate → browser hangs" report: the ad-panel InstancedMesh is transparent + DoubleSide + depthWrite:false and frustumCulled=false, with no distance/size LOD. Every media panel (media_count × 4) draws every frame; zoomed far out they all collapse into a small screen region and the transparent overdraw stalls the GPU, and rotating forces continuous re-render so it never recovers. Ruled out the picker (hover raycast is suppressed while the camera moves) and the fader (event-driven, zero per-frame cost) — the only image-scaling per-frame cost is this render. Fix: InstancedAdPanels.updateLOD(camera, viewportHeightPx), called once per frame from the buildings tick. It estimates the panels' on-screen height from the panel-cloud AABB + camera and hides the whole mesh (zero draw) once panels are sub-pixel, re-showing above a higher threshold (hysteresis, no flicker). O(1) per frame. Validation: adPanels.test.ts asserts the toggle (hidden zoomed-out, visible close, no-op at unknown viewport); largeRepoProfile.bench shows PostHog-scale zoom-out now draws 0 / 8192 panel instances (was all every frame). Owner to confirm the on-zoom-out smoothness in the browser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
At Linux scale the picker's THREE.Raycaster.intersectObjects tested every one of ~80k building instances on every pointer move (~33ms/cast — sub-30fps hovering, and each orbit/pan drag re-picks). The building geometry is a shared 12-triangle box, so a per-geometry BVH (three-mesh-bvh's acceleratedRaycast) does nothing here — measured 33→30ms. The cost is instance COUNT, so index the instances themselves: three-mesh-bvh's ObjectBVH treats each InstancedMesh instance as a BVH primitive. Measured 32.9 → 0.077ms/cast at 80k (~430x), one ~170ms build. The BVH is built lazily on the first pickAt after a refresh: keeps the build off the manifest-apply path and guarantees the object world matrices it reads are already fresh from the frame loop. Invalidated on every pickables refresh (cityRevision / decorationRevision). picker-bvh.test.ts guards equivalence: pickAt through the BVH returns the same object + instanceId + point + distance a brute-force core raycast would, against a real cell scene. largeRepoProfile.bench reports both paths side by side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real root cause of the media-heavy load hang (confirmed via the GitHub API: posthog/posthog ships ~1094 PNG + ~290 SVG ≈ 1.4k billboards): cellAssembly fired asyncLoadMediaForBuilding for EVERY media building synchronously on rebuild, regardless of visibility. So once the world painted, ~1.4k fetches + base64 decodes + canvas scales + copyTextureToTexture uploads drained in bursts on the main thread, janking every zoom/rotate until they finished. Defer + gate loading by visibility instead: - registerMediaBuilding now only QUEUES a load (records the panel center); cellAssembly no longer loads eagerly. - updateLOD (already per-frame) starts loads only for panels inside the camera frustum, at most AD_LOAD_BUDGET_PER_FRAME (4) per frame, and none at all while the panels are LOD-hidden (zoomed out). Off-screen panels stream in as the user rotates them into view. Behavior change to confirm in-browser: billboards now populate as you look around / zoom in rather than all at once on load (on PostHog's initial fit framing they may start blank until you zoom in — which is what removes the load hang). Tunable via AD_LOAD_BUDGET_PER_FRAME + the LOD px thresholds. Tests: adPanels.test.ts covers no-eager-load, frustum gating, per-frame budget draining without double-starts, and no loads while zoomed out. The load starter is injected so scheduling is observable without touching the network. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The whole-mesh visibility toggle was too coarse: at a mixed/medium zoom the foreground keeps the mesh visible, so every panel still renders — including the tiny unreadable background billboards that are pure overdraw (still laggy, per owner). Cull per building instead: each frame, collapse a building's 4 panels to zero-scale (a degenerate quad → no fragments) when it projects below AD_CULL_HIDE_PX or leaves the frustum, and restore above AD_CULL_SHOW_PX (hysteresis). Loading uses the same gate — a panel too small to read never fetches/decodes/uploads. The whole-mesh toggle stays as a cheap early-out when even the nearest panel is a speck (zoomed all the way out). Real matrices are cached per building so a culled panel restores without recompute. Cost is ~1 distance + frustum test per building per frame (only while the mesh is visible). adPanels.test.ts adds a mixed-scene guard: a close foreground panel renders + loads while a distant in-frustum one is collapsed to zero-scale + never loads. (Note: THREE's Matrix4.decompose reports [1,1,1] for an all-zero matrix, so the test checks the X-axis column length directly to detect the collapse.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The world-bbox computed expanded the Box3 with a freshly allocated THREE.Vector3 (and a rectOfStreet object) per corner — ~190k allocations over every street + building at Linux scale (issue #75 ranked bottleneck #4). Replace with direct min/max on box.min/max, inlining rectOfStreet's orientation swap. Bit-identical (guarded by cityState.test.ts; verified equal to the old expandByPoint output on an 80k layout) and ~2.4x faster with zero allocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…st (#75) The layout worker was posted the entire Manifest, whose commits array runs 100k–1M entries at Linux scale — and postMessage structured-clones it on the MAIN thread every apply. layoutCity reads only manifest.tree + manifest.stats (never commits), so that clone is pure waste: measured ~240ms for a 200k-commit manifest vs ~65ms for the {tree, stats} slice, and it scales with commit count (exactly the issue's "80k files AND 100k+ commits" case). compute() now posts only { tree, stats }; LayoutRequest.manifest is typed to a new LayoutManifest = Pick<Manifest,'tree'|'stats'>. This is the real worker-payload win (issue ranked bottleneck #1) — the geometry-only RETURN payload I also explored nets only ~10-25ms after re-association, so it's not worth the protocol churn. slimManifest.test.ts guards that layoutCity({tree,stats}) is byte-identical to layoutCity(fullManifest), so dropping commits can never change the city. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…size (#75) The per-instance cull gated on each panel's OWN projected height, so on one row at the same distance a small file's short billboard fell below the threshold and got culled/not-loaded while its taller neighbor loaded — a finicky mixed row, and it skipped panels the user was zoomed in on. Gate on a single reference height (the tallest panel in the repo) instead, so every panel at a given distance decides together: a row loads or culls as a unit. Also drop the aggressive per-instance thresholds (28/44 px) back to the whole-mesh ones (6/12), erring toward showing — a zoomed-in row now fully loads; only genuine far-background specks (sub-pixel) cull. Frustum + per-frame budget still bound loading to what's on-screen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An A/B switch to isolate whether the media billboards are the perf cost: under Buildings → Ad panels, "Enabled" (default on, Rebuild route). When off, cellAssembly skips the InstancedAdPanels entirely — no mesh, no fetch/decode/ upload, no draw, no per-frame LOD — while the media buildings themselves still render. Toggling re-applies via the existing Rebuild reaction (same route as the other AD_* knobs), so it recreates the cells with or without panels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A large media building at the screen edge could have its center just off-frame while its geometry clearly filled a corner — containsPoint(center) then culled it and never loaded its image (the front-right building in the report). Test the building's bounding sphere against the frustum instead, so an on-screen-but- off-center building loads and renders. Guarded by a test where the center fails containsPoint but the sphere rescues it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tick() walked all ~12k street labels every frame to orient them toward the camera (~4.2ms/frame at PostHog scale). But every X-oriented label flips together (all keyed off the camera's world-right X), likewise Y-labels off Z — so the flip is two booleans, not per-label. Track them, and only walk the label meshes when a boolean actually crosses its ±0.15 hysteresis threshold (rare) or after a rebuild swaps in fresh labels. Common frame is now O(1). Bit-identical flip behavior (guarded by the existing hysteresis test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Profiling PostHog-scale (ads OFF, so not the billboards): the base scene submits ~28.7k draw calls/frame, and streets are the bulk — every one of ~8.5k streets was TWO separate meshes (sidewalk + asphalt). Asphalt is a single solid color and never picked, so bake every street's asphalt pill into ONE merged geometry: ~8.5k asphalt draws → 1. Sidewalks stay per-street (they're the pickable directory targets + individually tinted) — merging those is a follow-up. streets.ts now exposes createSidewalkMesh + createMergedAsphaltMesh + capStyleFor (was the monolithic createStreetMesh); the component builds sidewalks per street and one asphalt mesh, disposes them individually, and the theme effect recolors the single asphalt material. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Street labels were ~11.7k separate transparent meshes at PostHog scale — each a draw call plus a per-frame transparent-depth sort, and the whole set enters the frustum when you zoom out (the lag case). The smallest-tier (leaf-directory) streets are ~35% of them, and their labels are unreadable specks exactly when zoomed out. Skip labels on the smallest-width streets once a city exceeds LABEL_LOD_MIN_STREETS (500) — 11.7k → 7.6k planes here; smaller repos keep every label. Behavior change worth an eyeball: deep leaf-dir streets no longer carry a name on big repos. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the street batching. Sidewalks were the other ~8.5k per-street draw calls, but unlike asphalt they're pickable AND individually tinted, so a plain merge isn't enough. The merged sidewalk carries a per-vertex color attribute (hover/select recolor only the hovered street's vertex span, with a partial GPU upload via addUpdateRange) and a faceIndex→street map on userData; the picker resolves an ObjectBVH raycast hit's faceIndex back to its directory (sidewalkStreetForFace, binary search). _sameHover now compares directories by dir.path since the mesh is shared. Net across the whole streets batch: PostHog-scale draw calls drop from ~28.7k to ~7.8k (streets ~16.9k → 2). Remaining draws are labels; a label atlas is the next lever if needed. Guards: mergedSidewalk.test (face→street mapping + contiguous ranges) and a picker-bvh case driving the full ObjectBVH → faceIndex → interpretHit chain to the correct dir. All 2580 unit tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
) The build-time leaf-dir label skip removed labels permanently, so zooming right up to a small street showed no name (owner report). Same lesson as the ad panels: cull by on-screen size per frame, don't drop at build time. Build every label again, and in tick() hide the ones that project below ~7px (re-show above ~12px, hysteresis) — up close a label is tens-to-hundreds of px tall and always shows; only far-away specks (the zoomed-out draw-call cost) get culled. The recompute runs only when the camera moves (else O(1)/frame), and doesn't cost more memory than before since all labels existed pre-change anyway. Guarded by a test: a label hides at a far camera and re-shows up close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Labels were still culling too eagerly at a normal zoom. Base the size estimate on the street's NATURAL label height (so a long directory name on a narrow street isn't culled just for being text-shrunk), and drop the thresholds to 3px hide / 6px show. Anything you're plausibly reading now stays visible; only far-away specks get culled when zoomed out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ound (#75) applyManifest awaited buildIconAtlas before setting any signal — and the atlas fetches one SVG per unique file extension from a CDN and awaited them ALL before returning. On an extension-diverse repo like PostHog that's a multi-second burst of network fetches blocking the ENTIRE city from rendering (the "empty during Sketching layout" gap). The iconName→slot UV map is just an index walk, so assign it up front and return the atlas synchronously; fetch + draw each SVG in the background, flagging the CanvasTexture for re-upload as each lands. Buildings bake correct roof UVs immediately and the icons pop onto already-built roofs progressively. A failed fetch just leaves its slot blank (UV stays mapped, so roofs never reshuffle). Bonus: applyManifest no longer awaits icon <img> loads, which never fire in jsdom — so tests that apply a real manifest can't hang on the atlas. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…until the stream ends (#75) The overlay hid when SCAN_PROGRESS cleared — which the stream's finally does the moment the final manifest is received. But setManifest only KICKS OFF applyManifest; its layoutCity runs async for a second-plus on a big repo, so the overlay vanished while the city was still assembling → the "empty 3D world on load" the owner saw on PostHog. loadingReactions now also watches REBUILD_STATUS: once the stream ends, it holds the overlay on "Building city" while the apply is still Rebuilding, and hides it only when the status leaves Rebuilding (Decorating/Idle) — i.e. when the buildings+streets are actually on screen. Settings rebuilds (no stream) still don't raise the overlay (the footer owns that). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gress (#75) A mid-transfer network drop on a huge repo (torvalds/linux) surfaced the ENTIRE git --progress stream as the error — thousands of "Receiving objects: N%" lines burying the actual failure. Two fixes in clone.py: - Classify the interruption: early EOF / RPC failed / unexpected disconnect / invalid index-pack output → a new CloneInterruptedError with a clean, retryable message ("interrupted mid-download … please try again"). The router already forwards str(e), so no routing change. - Strip the progress spam from every generic clone-failure message (_clean_git_stderr): drop the Receiving/Resolving/Enumerating/Counting lines, keep the last ~12 meaningful (error:/fatal:) lines. So even an unclassified failure never dumps a wall of percentages. Tests: the real linux failure tail classifies to CloneInterruptedError; the progress-stripper keeps the error lines and drops the percentages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… stream resets (#75) Root cause of the torvalds/linux failure: GitHub serves git over HTTP/2, and its stream multiplexing intermittently RSTs the pack transfer on very large repos ("curl 92 … HTTP/2 stream … CANCEL (err 8)" → fetch-pack disconnect → early EOF). ~1.4 GiB into an 8.5M-object clone, the stream got canceled and git aborted. Fix (_run_net_git wraps clone + fetch): - Force HTTP/1.1 (`-c http.version=HTTP/1.1`): a single unmultiplexed stream that doesn't hit the reset; throughput is equivalent for a one-pack clone. This is the actual cause fix. - Retry a transient drop up to 3x with backoff, cleaning the half-written clone between tries (git clone can't resume into it). Non-network failures (auth, not-found, branch) still fail fast; cancellation short-circuits. Tests: retry-then-succeed forces HTTP/1.1 on every attempt; a non-network error isn't retried; before_retry cleanup runs between attempts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A long recents list grew the landing page and scrolled the whole thing. Cap the action column to the viewport (minus the inner padding) so the "Open a project" card stays put and the recents list grows into the remaining height, then scrolls its own rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #75
Architectural perf pass for large repos (Linux-scale 80k files / 100k+ commits, and image-heavy repos like PostHog), driven by a headless per-phase profiler rather than guessing. Each change is justified by a before/after number and guarded by a test (or a bit-identical assertion).
Render / interaction
intersectObjectstested every one of ~80k building instances per pointer move (~34 ms/cast). three-mesh-bvh'sObjectBVH(each instance is a BVH primitive) drops it to ~0.077 ms/cast (~430×). Equivalence-guarded against a brute-force raycast.Vector3allocations (bit-identical).Media / billboards
Layout / worker
layoutCityreads onlytree+stats, so it's sent just that (~65 ms). Byte-identical layout guarded.Load experience
applyManifestawaited one CDN SVG fetch per extension before rendering anything. Now the UV map is assigned synchronously and icons fill the texture in the background, so the city renders without waiting on the fetch burst.layoutCitywas still running async → an empty 3D flash on big repos. Now held until the city actually renders.Clone (hit while testing torvalds/linux)
curl 92 … CANCEL→ early EOF). Clone/fetch now force HTTP/1.1 and retry a transient drop.Receiving objects: …%progress stream; it's classified and stripped to the real error.Profiler
app/bench/largeRepoProfile.bench.test.tstimes each apply phase for both workloads, with the picker core-vs-BVH and media-LOD numbers as regression guards. (Note: the street-label phase's absolute ms is jsdom-canvas-inflated and not browser-representative — flagged in the file.)All unit tests green (2583 frontend), backend suite green in docker (287).
🤖 Generated with Claude Code