docs: update architecture docs to match current codebase#361
Open
docs: update architecture docs to match current codebase#361
Conversation
- packages.md: add missing packages (logger, openapi, hono-zod-openapi, test-utils, dashboard, visualizer), fix stale dependency claims (yesql removed, stripe SDK → undici, protocol deps updated) - service/ARCHITECTURE.md: rewrite for Temporal-based architecture - engine/ARCHITECTURE.md: fix stale export names (forward→pipe, etc.) - principles.md: add logger+openapi to approved shared utilities - AGENTS.md: update package table and isolation rule - Remove deprecated docs (impl-status.md, overview.md) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Committed-By-Agent: claude
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Committed-By-Agent: claude
The knowledge-transfer deck had drifted away from the current engine, webhook, and connector behavior. This updates the slides to match the actual code paths, adds concrete protocol examples, and marks deprecated protocol message wrappers so the presentation teaches the current direction instead of legacy shapes. Constraint: Slides needed to stay faithful to current code paths while remaining presentation-readable Rejected: Keep generic interface aliases and legacy protocol examples | too ambiguous for a codebase handoff Rejected: Present webhook flow through Temporal/liveLoop | newer /pipeline_handle_events direction is the simpler story to teach Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep slides aligned with the actual protocol and API surfaces; if the message union or webhook entrypoints change again, update the deck at the same time Tested: Built Slidev deck in Docker; verified preview served over HTTP on localhost:3030 Not-tested: Runtime behavior of the deprecated protocol annotations beyond type/docs usage Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
The protocol annotation updates changed generated schema descriptions for deprecated eof payloads. The pre-push hook rejects drift, so this records the regenerated engine and service OpenAPI artifacts alongside the slide/protocol update already on the branch. Constraint: Repo pre-push hook requires generated OpenAPI files to match current source annotations Rejected: Push docs/protocol changes without regenerating specs | hook blocks on generated drift Confidence: high Scope-risk: narrow Reversibility: clean Directive: Any protocol description or schema metadata change that feeds OpenAPI should be followed immediately by ./scripts/generate-openapi.sh Tested: Ran ./scripts/generate-openapi.sh and inspected generated engine/service OpenAPI diffs Not-tested: Full downstream client compatibility beyond generated type refresh Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
tonyxiao
changed the title
There was a problem hiding this comment.
Pull request overview
Updates the repository’s architecture documentation to reflect the current Temporal-based service + engine design, refreshes the monorepo package/dependency graph, and adds/adjusts Slidev-based knowledge-transfer slides (including Docker build/runtime updates).
Changes:
- Rewrites architecture docs (
docs/architecture/*,docs/service/*,docs/engine/*,AGENTS.md) to align with current packages, exports, and Temporal orchestration. - Adds a new Slidev knowledge-transfer deck + outline and tweaks Slidev styling and Docker build images.
- Updates protocol/OpenAPI wording around legacy EOF-related schemas/messages and aligns a workflow failure check with the newer EOF status field.
Reviewed changes
Copilot reviewed 15 out of 19 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/protocol/src/protocol.ts | Adds deprecation annotations / updated descriptions for legacy terminal/push-input message types. |
| docs/slides/style.css | Adjusts mermaid sizing and enables vertical scrolling for overflow content. |
| docs/slides/package.json | Adds a new script to run the knowledge-transfer slide deck. |
| docs/slides/knowledge-transfer.md | New knowledge-transfer slide deck describing protocol, components, and deployment surfaces. |
| docs/slides/knowledge-transfer-outline.md | New outline document for the knowledge-transfer deck. |
| docs/slides/Dockerfile | Bumps slide build/runtime Docker base images to Node 24. |
| docs/service/scenarios.md | Updates referenced “key file” from legacy service class to store interfaces. |
| docs/service/overview.md | Removes deprecated/obsolete service overview doc. |
| docs/service/ARCHITECTURE.md | Rewrites service architecture section for Temporal-based orchestration and current file layout. |
| docs/engine/ARCHITECTURE.md | Updates docs to current engine exports/util names (pipe, filterType, etc.). |
| docs/architecture/principles.md | Updates connector isolation rule text to include approved shared utilities. |
| docs/architecture/packages.md | Major rewrite of package inventory + dependency layering to match current monorepo. |
| docs/architecture/impl-status.md | Removes deprecated/obsolete implementation status doc. |
| apps/service/src/temporal/workflows/pipeline-backfill.ts | Updates terminal failure detection to use eof.status. |
| apps/service/src/generated/openapi.json | Regenerates OpenAPI with updated EOF schema descriptions. |
| apps/service/src/generated/openapi.d.ts | Regenerates OpenAPI types with updated EOF schema descriptions. |
| apps/engine/src/generated/openapi.json | Regenerates OpenAPI with updated EOF schema descriptions. |
| apps/engine/src/generated/openapi.d.ts | Regenerates OpenAPI types with updated EOF schema descriptions. |
| AGENTS.md | Updates the package table + isolation wording to match current packages/deps. |
Files not reviewed (2)
- apps/engine/src/generated/openapi.json: Language not supported
- apps/service/src/generated/openapi.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
436
to
460
| @@ -451,8 +452,11 @@ export const EofPayload = z | |||
| ), | |||
| request_progress: ProgressPayload.describe('Progress for this specific request only.'), | |||
| }) | |||
| .describe('Terminal message signaling end of this request.') | |||
| .describe( | |||
| 'Deprecated terminal message signaling end of this request. Prefer explicit request/response results via pipeline_sync_batch.' | |||
| ) | |||
| .meta({ id: 'EofPayload' }) | |||
| /** @deprecated Legacy terminal payload. Prefer explicit request/response results via pipeline_sync_batch. */ | |||
| export type EofPayload = z.infer<typeof EofPayload> | |||
Comment on lines
+5
to
+6
| - as a library (replit, supabasea) | ||
| - as cli (direct users, contributors) |
Comment on lines
+29
to
+33
| - source-stripe | postgres destiination | ||
| - Putting it all together | ||
| - scheduled sync | ||
| - start new sync run | ||
| - 1000 state messages per pipeline_sync requedst |
| - real time | ||
| - `/pipeline_handle_events` | ||
| - Experimental | ||
| - When we started the sync engine, the vision is to become an ubiquitous utility to help user “get your data where you need it, in real time, with a consistent schema” |
| const engine = await createEngine(resolver) | ||
| const eof = await engine.pipeline_sync_batch(pipeline, { run_id: 'run_demo' }) | ||
| ``` | ||
| - as cli |
|
|
||
| ## Experimental | ||
|
|
||
| - When we started the sync engine, the vision is to become an ubiquitous utility to help user “get your data where you need it, in real time, with a consistent schema” |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Committed-By-Agent: claude
Records must be nested as `{"type":"record","record":{...}}` per the
RecordMessage schema, and the catalog needs `newer_than_field` with a
matching timestamp in record data for destination-postgres upserts.
Also adds dummy-source demo scripts and reorders KT slides.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
This pass simplifies the opening slide, tightens the webhook and subdivision explanations, and adjusts layout choices so the deck behaves better in Slidev during an actual presentation rather than just reading well in source form. Constraint: Slides need to fit the live Slidev rendering model, not just look correct in markdown Rejected: Leave the slide structure as-is and rely on browser refreshes | rendering/layout issues were slide-source problems, not just stale client state Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep presentation changes grouped and verify them in the running Slidev preview before committing Tested: Rebuilt Slidev deck in Docker repeatedly and verified preview responses on localhost:3030 Not-tested: Remote browser rendering differences beyond the local preview Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StatefulSyncclass andapps/statefulpath)forward→pipe,filterDataMessages→filterType)loggerandopenapito approved shared utilities listimpl-status.md,service/overview.mdHow to test (optional)
pnpm buildpassesgrep -r "yesql\|StatefulSync\|apps/stateful" docs/returns nothing in active docs