Fix reverse ETL progress and connector schemas#362
Open
kdhillon-stripe wants to merge 2 commits intomainfrom
Open
Fix reverse ETL progress and connector schemas#362kdhillon-stripe wants to merge 2 commits intomainfrom
kdhillon-stripe wants to merge 2 commits intomainfrom
Conversation
Committed-By-Agent: cursor
There was a problem hiding this comment.
Pull request overview
This PR improves reverse ETL behavior by (1) making Postgres source runs report stream lifecycle completion even when no rows are read, (2) ensuring connector config JSON Schemas that are not plain objects (e.g., anyOf unions) are preserved for validation and OpenAPI generation, and (3) registering reverse ETL connectors + scoping CLI connector shorthand flags by side to avoid name collisions.
Changes:
- Emit
stream_status: start/completefor Postgres source reads (including empty incremental reads) so no-op runs persist as completed. - Preserve non-object connector JSON Schemas across engine/service schema factories and resolver validation.
- Add Postgres source + Stripe destination to service CLI/OpenAPI generation, and update CLI shorthand flags to
--source.<connector>.*/--destination.<connector>.*.
Reviewed changes
Copilot reviewed 15 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/generate-openapi-specs.ts | Includes Postgres source + Stripe destination in generated Engine/Service OpenAPI specs. |
| pnpm-lock.yaml | Locks workspace deps for newly referenced connectors. |
| packages/source-postgres/src/index.ts | Emits stream_status lifecycle events around Postgres reads. |
| packages/source-postgres/src/index.test.ts | Updates expectations and adds coverage for empty-stream completion behavior. |
| e2e/reverse-etl-demo-loop.ts | Extends demo loop with a new product pipeline/table for reverse ETL. |
| apps/service/src/lib/createSchemas.ts | Preserves non-object connector schemas when building service request/response schemas. |
| apps/service/src/lib/createSchemas.test.ts | Adds regression test ensuring anyOf connector config fields are not stripped. |
| apps/service/src/lib/cli-connector-shorthand.ts | Scopes shorthand flags by side and updates dotted-flag validation logic. |
| apps/service/src/lib/cli-connector-shorthand.test.ts | Updates tests for side-scoped shorthand and overlapping connector names. |
| apps/service/src/cli.ts | Registers Postgres source + Stripe destination in the service CLI resolver. |
| apps/service/src/cli.test.ts | Updates generated CLI tests to use --source.* / --destination.* shorthand. |
| apps/service/src/generated/openapi.json | Regenerates service OpenAPI with Postgres source + Stripe destination schemas. |
| apps/service/src/generated/openapi.d.ts | Regenerates service OpenAPI TypeScript types for new/updated schemas. |
| apps/service/package.json | Adds workspace deps for @stripe/sync-source-postgres and @stripe/sync-destination-stripe. |
| apps/engine/src/lib/resolver.ts | Preserves non-ZodAny schemas from connector specs for config validation. |
| apps/engine/src/lib/resolver.test.ts | Adds test ensuring union config schemas remain validating (not collapsed to {}). |
| apps/engine/src/lib/createSchemas.ts | Preserves non-object connector schemas when building engine pipeline schemas. |
| apps/engine/src/lib/createSchemas.test.ts | Adds regression test ensuring anyOf connector config fields are not stripped. |
| apps/engine/src/generated/openapi.json | Regenerates engine OpenAPI with Postgres source + Stripe destination schemas. |
| apps/engine/src/generated/openapi.d.ts | Regenerates engine OpenAPI TypeScript types for new/updated schemas. |
Files not reviewed (3)
- apps/engine/src/generated/openapi.json: Language not supported
- apps/service/src/generated/openapi.json: Language not supported
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+237
to
240
| yield msg.stream_status({ stream: name, status: 'start' }) | ||
|
|
||
| const pool = await createPool(config) | ||
| try { |
Comment on lines
+156
to
160
| for (const side of ['source', 'destination'] as const) { | ||
| for (const [connectorName, config] of grouped[side]) { | ||
| result[side] = { type: connectorName, [connectorName]: config } | ||
| } | ||
| } |
Fix source-postgres connector specification so conformance sees a valid JSON Schema shape for `config`, prevent destination-stripe from forwarding non-error stream status once a stream has failed, and make the reverse ETL custom-object double-run test state progression deterministic. Committed-By-Agent: cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
not_startedprogress.Test plan
./scripts/generate-openapi.shsource scripts/prefer-system-esbuild.sh && pnpm --filter @stripe/sync-source-postgres exec vitest run src/index.test.tssource scripts/prefer-system-esbuild.sh && pnpm --filter @stripe/sync-engine exec vitest run src/lib/createSchemas.test.ts src/lib/resolver.test.tssource scripts/prefer-system-esbuild.sh && pnpm --filter @stripe/sync-service exec vitest run src/lib/createSchemas.test.ts src/lib/cli-connector-shorthand.test.ts src/cli.test.tspnpm lintsource scripts/prefer-system-esbuild.sh && pnpm buildNote:
pnpm format:checkstill reports unrelated existing formatting issues outside this PR; PR-owned files were formatted directly.