UX-1338 — Redpanda Connect Pipeline (RPCN) Visual View & Editor

[SpicyPete]

Summary

Adds a full visual editor for Redpanda Connect (RPCN) pipelines: an interactive, node-based canvas that renders a pipeline's YAML as a directed graph and lets you inspect and edit nodes without hand-writing YAML. It sits alongside the existing YAML editor as a new "Visual" lane.

The editor is feature-flagged and currently scoped to the embedded (Cloud) context — it activates only when enableRpcnVisualEditor and enablePipelineDiagrams are on and the page is embedded. When enabled, editing opens on the Visual lane by default; otherwise the experience is unchanged (YAML lane).

What's included

Canvas & layout

• Flow canvas (pipeline-flow-canvas.tsx, pipeline-flow-canvas-nodes.tsx) — renders the pipeline as a left-to-right DAG using React Flow + Dagre (@dagrejs/dagre, new dependency): input → processors → output, with control-flow constructs (switch, branch, try/catch, broker, fallback) drawn as split → case-lanes → merge, plus conditional/error/reference edge styling.
• Scope regions — each control-flow construct gets a faint dashed enclosure. These are traced as a single connected outline that hugs the construct's actual (L-shaped) footprint per Dagre column, so a fan-out no longer paints over unrelated neighbor cards. Tall regions tile their fill under the browser's max paintable-element size so the background never silently drops.
• Flow parser & layout (pipeline-flow-parser.ts, pipeline-flow-meta.ts) — parses pipeline YAML into a flow tree and computes the layout with source-aligned ranks. Replaces the old sidebar diagram (pipeline-flow-diagram.tsx removed).

Editing

• Node inspector (node-inspector.tsx, node-config-form.tsx) — side panel to view/edit a selected node's config via Registry Field forms (react-hook-form + Zod), drill into control-flow children (switch cases / steps), and edit a node's raw YAML in place.
• Editor store (use-pipeline-editor-store.ts) — Zustand store with undo/redo; pending visual edits are flushed to YAML at save time.
• Component picker / command palettes (pipeline-canvas-command-palette.tsx, onboarding/connect-command-palette.tsx, component-aliases.ts) — quick actions (view in YAML, undo/redo, jump to node) and a connector/processor inserter with curated defaults, alias-aware search (e.g. "queue" → kafka_franz), and de-emphasis of deprecated/experimental components.

Feedback surfaces

• Structure tree (pipeline-structure-tree.tsx) — sidebar outline in document order with collapsible groups, roving-tabindex keyboard navigation, and click-to-reveal.
• Problems panel (pipeline-problems-panel.tsx, pipeline-lint.ts) — floating chip listing lint problems (mapped from backend LintHints by YAML line range, merged with save-error hints) and detected secret references; click a problem to jump to its node.
• Unsaved panel (pipeline-unsaved-panel.tsx, pipeline-diff.ts) — floating chip listing nodes whose config differs from the last save (signature-based diff); click to jump.
• Tips bar & template CTA (editor-tips-bar.tsx, template-cta.tsx) — platform-aware keyboard-shortcut hints and a "start from a template" entry point for empty pipelines.

Testing

Unit + integration tests accompany the new modules — flow parser/layout, lint mapping, dirty-diff, canvas render (incl. tall-region paint-cap), inspector, config form, structure tree, problems/unsaved panels, and command palettes. type:check, lint, and the test suite are green.

Known limitations / deferrals

• Delete-guard (confirming deletion of a node with dependents) is a TODO.

How to test

Enable enableRpcnVisualEditor + enablePipelineDiagrams, open a pipeline in the embedded editor, and confirm the Visual lane loads. Verify node selection/edit, undo/redo, save round-trips to YAML, and the problems/unsaved chips.

Screenshots

Sidelane / YAML enhancements

Visual view

[github-actions]

✅ Clean — no registry drift, off-token colours, or ad-hoc classes

App: frontend · Scope: diff vs origin/master · Files: 53

	Count
⚠️ Outdated registry components	0
🛠 Locally-modified components	0
❓ Unknown to registry	0
🎨 Off-token palette colours	0
🔢 Ad-hoc utility classes	0

Generated by lookout audit-changes.

[SpicyPete]

@claude review

[github-actions]

Claude finished @SpicyPete's task in 1m 24s —— View job

---

Code Review — UX-1338 RPCN Visual Editor

Reviewed across 5 parallel passes (parser/layout, canvas, inspector/store, yaml/utils, panels/shared-UI). This is a large but carefully written PR — effect cleanup, undo/redo, the save-flush path, and the shared-UI signature changes all check out. Below are the findings worth acting on, most impactful first.

---

🟠 Correctness

1. mergeScanner picks the wrong input key when the input has a label — utils/yaml.ts:134

const inputType = Object.keys(inputObj)[0];   // ← should be firstKey(inputObj)

Everywhere else in this file uses firstKey() (e.g. yaml.ts:534, :543) specifically to skip reserved keys (label, <<). Here the raw first key is used, so for input: { label: my_in, kafka_franz: {...} } the scanner gets written to input.label.scanner (line 142), corrupting the config instead of attaching to kafka_franz.
Repro: labeled input file → pick a scanner from the palette → scanner nested under the label scalar.
Fix this →

2. Switch-case condition draft leaks across sibling cases → silent misrouting — pipeline/node-inspector.tsx:510-522

const initial = typeof caseObject.check === 'string' ? caseObject.check : '';
const [check, setCheck] = useState(initial);
useEffect(() => setCheck(initial), [initial]);   // only resyncs when the primitive value changes

SwitchCaseEditor/CaseConditionSection isn't re-keyed across selection changes, so the instance persists. Select case A (empty check), type foo, then select sibling case B (also empty check): initial is still '' (unchanged), the resync effect never fires, check stays foo, dirty flips true, and a phantom draft caseWithCheck(B, 'foo') is reported. On the next flush/Save, B's routing condition is silently set to a value the user never entered for it.
Fix direction: key the editor by case/target identity, or resync on caseObject identity rather than the primitive initial.
Fix this →

3. multiMemberSpecs doesn't skip null/unparseable input entries → phantom leaf — utils/pipeline-flow-parser.ts:175-184
Unlike extractProcessorEntries (:473) and parseSwitchCases (:565), this maps every array element without filtering non-objects. For input: { broker: { inputs: [null, { kafka: {} }] } }, parseMultiInputs filters to one key but multiMemberSpecs returns two specs → a phantom input leaf whose editTarget points at the null slot (['input','broker','inputs',0]). Clicking/deleting it operates on the null entry. (Array-index alignment itself is correct — this is a spurious-node bug, not an off-by-one.)
Fix this →

---

🟡 Performance (large graphs)

4. Scope-region work re-runs on pan/zoom and hover — pipeline/pipeline-flow-canvas.tsx

• ScopeRegions useStore selector (:903-912) folds a hash over every node on each React Flow store mutation — including the transform updates emitted per animation frame during pan/zoom. On a few-hundred-node pipeline this is a full-node loop ~60×/s just to detect measurement changes.
• RegionBox (:820) calls regionGeometry(...) (which runs unionOutline — grid build + boundary chaining) in its render body, and ScopeRegions re-renders on any hoveredNodeId/selectedNodeId change (:956-967). So hovering any node re-traces the outline of every region — contradicting the "hover stays cheap" design intent. The regions memo caches columns but not the outline path.

Fix direction: derive measuredKey only from node dimensions (not viewport transform), and memoize RegionBox/the outline path so it recomputes only when its own geometry changes.
Fix this →

---

🟢 Minor / nits

• component-aliases.ts:4-47 — substring alias matching over-matches: bloblang contains log so searching "log" surfaces the bloblang mapping processor; the sql fragment tags any *sql* name with all three db aliases. Consider matching alias tokens on word boundaries rather than substring.
• Minimap a11y (pipeline-flow-canvas.tsx:195-245) — interactive <svg role="img"> with pointer handlers but no tabIndex/keyboard support, and no onPointerCancel (:199-212), so an interrupted drag can leave draggingRef.current === true and keep re-centering on subsequent moves.
• Sequence-input semantics ignored (pipeline-flow-parser.ts:1316-1321) — childFlow: 'sequential' is set at parse time but fanSides forces in: true for all input containers, so a sequence input renders identically to a broker (parallel fan-in) despite the different intended semantics.
• Switch-case labels skip numbers (pipeline-flow-parser.ts:573) — caseId/caseLabel use the raw array index, so a skipped/null case yields "case 1", "case 3" (no "case 2").
• connect-command-palette.tsx:447 — recents snapshot read once via useMemo(…, []); the "Recent" tab won't reflect a just-added component if the palette is reused without unmounting (harmless if it always unmounts on commit).
• Unstable onConfigChange deps (node-inspector.tsx:518, :786; node-config-form.tsx:871) — inline arrow in effect deps makes reporter effects run every render. Benign (writes only to refs) but wrap in useCallback for clarity.
• EMPTY_CONFIG_NODES (pipeline-flow-parser.ts:1036) — module-level array returned by reference for empty configs; safe today but a latent shared-mutable-state footgun.
• tabs.tsx:5 — import React used only as a type (React.ComponentProps); import type would be cleaner (won't fail CI — redpanda-ui is lint-excluded).

Also worth noting the PR's own stated deferral: the delete-guard for nodes with dependents is still a TODO, and countResourceReferences (yaml.ts:1235) only counts explicit resource: fields — it can under-warn vs. the flow parser's usedByCount if it ever feeds that prompt.

---

Nothing here is a merge-blocker for a feature-flagged surface, but I'd fix #1 and #2 before wider rollout — both silently corrupt/misroute saved config. Nice work on the parser test coverage and the paint-cap tiling.

[SpicyPete]

#2554
This PR pulls out a bunch of the non-core updates to this PR