Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,25 @@ A Google Calendar import into Compass.
A Google sync recovery path that refreshes or restarts Google data when metadata
says the existing sync needs repair.

**Google Watch repair**:
A repair path that recreates missing, expired, stale, or incomplete **Google
Watches** for a **Google-connected user**. Repair should happen only when watch
state is broken; regular health checks should not repeatedly call Google when
watches are already healthy. When sync tokens are usable, watch repair should
also catch up missed Google-side changes through incremental import; when sync
tokens are missing or invalid, fall back to full **Repair**.

**Public watch notifications**:
Google-to-Compass webhook posts at `/api/sync/gcal/notifications`.

**Compass-to-Google event propagation**:
The backend flow that applies Compass Event writes and mirrors eligible Google
side effects.

**Google-to-Compass event propagation**:
The backend flow that applies Google Calendar event changes to Compass data
during Import or Public watch notification handling.

## Relationships

- An **Anonymous user** stores **Events** and **Tasks** in browser IndexedDB.
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/event-and-task-domain-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ If you change recurring edit behavior, check:

- `packages/core/src/types/event.types.ts`
- `packages/backend/src/event/controllers/event.controller.ts`
- `packages/backend/src/sync/services/sync/compass/compass.sync.processor.ts`
- `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google.event-propagation.ts`

## Backend Event Shape Semantics

Expand Down
12 changes: 7 additions & 5 deletions docs/backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,24 +49,26 @@ Important runtime behavior:

## Event Writes

- Event writes route through the Compass sync processor and can apply Google side effects.
- Event writes route through Compass-to-Google event propagation and can apply Google side effects.
- Missing Google refresh token does not block Compass-local event writes; Google side effects are skipped.
- Controllers use the shared `res.promise(...)` response helper and centralized
error handling.

Key files:

- `packages/backend/src/event/controllers/event.controller.ts`
- `packages/backend/src/sync/services/sync/compass/compass.sync.processor.ts`
- `packages/backend/src/sync/services/outbound/compass-google-mirror.service.ts`
- `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google.event-propagation.ts`
- `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google-backfill.ts`

## Google Notification Ingress

- endpoint: `POST /api/sync/gcal/notifications`
- source: `packages/backend/src/sync/controllers/sync.controller.ts`
- middleware: `authMiddleware.verifyIsFromGoogle`
- middleware: `publicWatchNotificationIngress.verify`
- ingress owner: `packages/backend/src/sync/services/public-watch-notifications/public-watch-notification.ingress.ts`
- notification owner: `packages/backend/src/sync/services/watch/google-watch.service.ts`
- import/repair owner: `packages/backend/src/sync/services/google-calendar-sync/google-calendar-sync.service.ts`
- repair/setup owner: `packages/backend/src/sync/services/google-sync/google-sync.service.ts`
- import owner: `packages/backend/src/sync/services/import/google-import.service.ts`
- maintenance owner: `packages/backend/src/sync/services/watch/google-watch-maintenance.service.ts`

Observed outcomes include:
Expand Down
7 changes: 4 additions & 3 deletions docs/backend/backend-request-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,9 @@ For `POST /api/event`:
1. route requires `verifySession()`
2. controller adds the authenticated user id
3. controller normalizes single vs array payloads
4. controller forwards the change set to `CompassSyncProcessor`
4. controller forwards the change set to `CompassToGoogleEventPropagation`
5. controller returns a status-only payload (`{ statusCode: 204 }`) through `res.promise(...)`
6. processor:
6. Compass-to-Google event propagation:
- loads current Compass DB state
- analyzes the transition into a persistence plan
- applies Compass DB mutations first
Expand All @@ -118,7 +118,8 @@ Frequently used middleware:
- `requireGoogleConnectionSession`: active Google connection required for routes that call Google directly (for example `/api/sync/import-gcal`)
- `authMiddleware.verifyIsDev`: development-only route
- `authMiddleware.verifyIsFromCompass`: trusted internal caller
- `authMiddleware.verifyIsFromGoogle`: trusted Google notification source
- `publicWatchNotificationIngress.verify`: trusted Google notification source
- `packages/backend/src/sync/services/public-watch-notifications/public-watch-notification.ingress.ts`: Public watch notification header validation and payload parsing

Intentional unauthenticated route:

Expand Down
6 changes: 3 additions & 3 deletions docs/development/common-change-recipes.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Rule: never treat event shape as web-only unless the field is strictly presentat
3. Read `packages/backend/src/event/classes/compass.event.generator.ts`.
4. Read `packages/backend/src/event/classes/compass.event.parser.ts`.
5. Read `packages/backend/src/event/classes/compass.event.executor.ts`.
6. Read `packages/backend/src/sync/services/sync/compass/compass.sync.processor.ts`.
6. Read `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google.event-propagation.ts`.
7. Update the planner, executor, or scope-expansion path that actually owns the behavior.
8. Add focused tests for the exact recurrence transition you changed.

Expand All @@ -43,13 +43,13 @@ Do not edit recurring behavior from one layer only.
## Triage A Recurrence Sync Regression

1. Reproduce with one event and one expected transition outcome.
2. Capture processor logs for the transition key:
2. Capture Compass-to-Google event propagation logs for the transition key:
- `Handle Compass event(<id>): <transitionKey>`
3. Find the key in `PLAN_BUILDERS` in `packages/backend/src/event/classes/compass.event.parser.ts`.
4. Confirm planned `steps` and `googleEffect` match expected behavior.
5. Confirm executor step mapping in `packages/backend/src/event/classes/compass.event.executor.ts`.
6. Run focused tests:
- `bun run test:backend --runTestsByPath packages/backend/src/event/classes/compass.event.parser.test.ts packages/backend/src/event/classes/compass.event.executor.test.ts packages/backend/src/sync/services/sync/__tests__/compass.sync.processor.test.ts --runInBand`
- `bun run test:backend --runTestsByPath packages/backend/src/event/classes/compass.event.parser.test.ts packages/backend/src/event/classes/compass.event.executor.test.ts packages/backend/src/sync/services/event-propagation/__tests__/compass-to-google.event-propagation.test.ts --runInBand`

## Add An SSE Event

Expand Down
10 changes: 6 additions & 4 deletions docs/development/feature-file-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,12 +82,14 @@ Use this document to find the first files to inspect for common Compass changes.
- Backend SSE server: `packages/backend/src/servers/sse/sse.server.ts`
- Events stream route: `packages/backend/src/events/events.routes.config.ts`
- Backend sync routes: `packages/backend/src/sync/sync.routes.config.ts`
- Public watch notification ingress: `packages/backend/src/sync/services/public-watch-notifications/public-watch-notification.ingress.ts`
- Google Watch lifecycle and notifications: `packages/backend/src/sync/services/watch`
- Google Calendar sync import and repair: `packages/backend/src/sync/services/google-calendar-sync/google-calendar-sync.service.ts`
- Compass-to-Google repair mirroring: `packages/backend/src/sync/services/outbound/compass-google-mirror.service.ts`
- Sync record persistence: `packages/backend/src/sync/services/records/sync.records.ts`
- Google Calendar sync import and repair: `packages/backend/src/sync/services/google-sync/google-sync.service.ts`
- Google sync health diagnosis: `packages/backend/src/sync/services/google-sync/google-sync.health.ts`
- Compass-to-Google repair mirroring: `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google-backfill.ts`
- Sync record persistence: `packages/backend/src/sync/services/records/sync-records.repository.ts`
- Google import internals: `packages/backend/src/sync/services/import`
- Google/Compass event processors: `packages/backend/src/sync/services/sync`
- Google/Compass event propagation: `packages/backend/src/sync/services/event-propagation`

## Users / Metadata / Priority

Expand Down
2 changes: 1 addition & 1 deletion docs/development/testing-playbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ Preferred style:
- realistic request flows when possible
- mock only external services, not internal business logic

**Do not import `mongoService` (or other persistence implementations) directly in tests.** Use test drivers instead (e.g. `UserDriver`, `WatchDriver` in `packages/backend/src/__tests__/drivers/`). Drivers encapsulate persistence so that switching away from Mongo (or another store) in the future does not require changing test code.
**Do not import `mongoService` (or other persistence implementations) directly in tests.** Use test drivers instead (e.g. `UserDriver`, `GoogleWatchDriver` in `packages/backend/src/__tests__/drivers/`). Drivers encapsulate persistence so that switching away from Mongo (or another store) in the future does not require changing test code.

Useful anchors:

Expand Down
2 changes: 1 addition & 1 deletion docs/development/types-and-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Use the generic base schema with `parseApiError(error, ApiErrorResponseSchema)`

- `packages/backend/src/common/constants/env.constants.ts`
- `packages/backend/src/common/validators`
- feature-specific query validators such as `packages/backend/src/sync/util/sync.queries.ts`
- feature-specific query validators such as `packages/backend/src/sync/services/records/sync-records.repository.ts`

### Web form/client validation

Expand Down
29 changes: 18 additions & 11 deletions docs/features/google-sync-and-sse-flow.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Google Sync And Server-Sent Events (SSE)

Compass sync is bidirectional:
Compass Google Calendar integration is bidirectional:

- Compass-originated event changes can propagate to Google and then notify web clients.
- Google-originated changes can flow back into Compass and then notify web clients.
Expand Down Expand Up @@ -78,7 +78,7 @@ Source:

- `packages/core/src/types/sse.types.ts`
- `packages/backend/src/user/services/user.service.ts`
- `packages/backend/src/sync/services/google-calendar-sync/google-calendar-sync.service.ts`
- `packages/backend/src/sync/services/google-sync/google-sync.service.ts`

`IMPORT_GCAL_END` carries an explicit `operation` so the client can distinguish repair completion from incremental completion.

Expand Down Expand Up @@ -114,41 +114,42 @@ High-level path:
3. The selected repository writes locally or remotely.
4. Remote event writes hit backend event routes.
5. `EventController` packages the change as a `CompassEvent`.
6. `CompassSyncProcessor.processEvents()` loads the DB event, plans work, applies persistence, and runs Google side effects.
6. `CompassToGoogleEventPropagation.processEvents()` loads the DB event, plans work, applies persistence, and runs Google side effects.
7. After commit, the backend calls `sseServer` to publish notifications based on whether the change affected normal or someday events (`EVENT_CHANGED` vs `SOMEDAY_EVENT_CHANGED`).

Primary files:

- `packages/web/src/ducks/events/sagas/event.sagas.ts`
- `packages/web/src/common/repositories/event`
- `packages/backend/src/event/controllers/event.controller.ts`
- `packages/backend/src/sync/services/sync/compass/compass.sync.processor.ts`
- `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google.event-propagation.ts`

## Inbound Flow: Google Notifies Compass About Changes

High-level path:

1. Google posts to the notification endpoint in sync routes.
2. Backend verifies the request origin.
2. `publicWatchNotificationIngress` validates and parses the Google headers.
3. `googleWatchService.handleGoogleWatchNotification()` locates the watch and sync record.
4. The service builds a Google Calendar client for the user.
5. `GCalNotificationHandler` fetches incremental changes using the stored sync token.
6. `GcalSyncProcessor` applies those changes to Compass data.
6. `GoogleToCompassEventPropagation` applies those changes to Compass data.
7. The backend publishes `EVENT_CHANGED` (or someday equivalent) so clients refetch.

Primary files:

- `packages/backend/src/sync/sync.routes.config.ts`
- `packages/backend/src/sync/services/public-watch-notifications/public-watch-notification.ingress.ts`
- `packages/backend/src/sync/services/watch/google-watch.service.ts`
- `packages/backend/src/sync/services/google-calendar-sync/google-calendar-sync.service.ts`
- `packages/backend/src/sync/services/records/sync.records.ts`
- `packages/backend/src/sync/services/google-sync/google-sync.service.ts`
- `packages/backend/src/sync/services/records/sync-records.repository.ts`
- `packages/backend/src/sync/services/notify/handler/gcal.notification.handler.ts`
- `packages/backend/src/sync/services/sync/google/gcal.sync.processor.ts`
- `packages/backend/src/sync/services/event-propagation/google-to-compass/google-to-compass.event-propagation.ts`

Lifecycle and outbound repair paths live in:

- `packages/backend/src/sync/services/google-calendar-sync/google-calendar-sync.service.ts`
- `packages/backend/src/sync/services/outbound/compass-google-mirror.service.ts`
- `packages/backend/src/sync/services/google-sync/google-sync.service.ts`
- `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google-backfill.ts`
- `packages/backend/src/sync/services/watch/google-watch-maintenance.service.ts`

### Notification Outcomes And Error Semantics
Expand Down Expand Up @@ -206,9 +207,15 @@ Redux reasons for refetch (`Sync_AsyncStateContextReason`) reuse the same string
Source files:

- `packages/backend/src/user/services/user-metadata.service.ts`
- `packages/backend/src/sync/services/google-sync/google-sync.health.ts`
- `packages/core/src/types/user.types.ts`
- `packages/web/src/sse/hooks/useGcalSSE.ts`

The sidebar Google connection state is derived from user metadata. The backend
metadata service delegates HEALTHY vs ATTENTION diagnosis to Google sync health,
which checks stored sync tokens and, when public HTTPS watch notifications are
enabled, active Google watches.

Auto-import guardrail:

- client auto-starts import only when `sync.importGCal === "RESTART"` **and** `google.connectionState` is not `NOT_CONNECTED` or `RECONNECT_REQUIRED`
Expand Down
4 changes: 2 additions & 2 deletions docs/features/password-auth-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,8 @@ Password-only users can now mutate Compass events without a Google connection at
Relevant changes:

- `event.routes.config.ts` no longer requires route-level Google connection middleware for create/update/delete
- `CompassSyncProcessor` applies the Compass mutation first
- if the Google side effect fails only because the user has no Google refresh token, the processor keeps the Compass mutation and skips the Google effect
- `CompassToGoogleEventPropagation` applies the Compass mutation first
- if the Google side effect fails only because the user has no Google refresh token, Compass-to-Google event propagation keeps the Compass mutation and skips the Google effect

That lets password-auth users use Compass without blocking on Google connectivity.

Expand Down
18 changes: 9 additions & 9 deletions docs/features/recurring-events-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Primary files:

## Recurrence Categories

Compass sync logic classifies event shape using `Categories_Recurrence`:
Compass-to-Google event propagation classifies event shape using `Categories_Recurrence`:

- `STANDALONE`
- `RECURRENCE_BASE`
Expand All @@ -28,7 +28,7 @@ Compass sync logic classifies event shape using `Categories_Recurrence`:
- `RECURRENCE_BASE_SOMEDAY`
- `RECURRENCE_INSTANCE_SOMEDAY`

The Compass sync path treats recurrence handling as a transition problem:
The Compass-to-Google path treats recurrence handling as a transition problem:

1. build a transition context from the incoming Compass payload plus the current DB event
2. analyze that transition into a plain `CompassOperationPlan`
Expand All @@ -39,7 +39,7 @@ Primary files:

- `packages/backend/src/event/classes/compass.event.parser.ts`
- `packages/backend/src/event/classes/compass.event.executor.ts`
- `packages/backend/src/sync/services/sync/compass/compass.sync.processor.ts`
- `packages/backend/src/sync/services/event-propagation/compass-to-google/compass-to-google.event-propagation.ts`

## Update Scopes

Expand Down Expand Up @@ -95,7 +95,7 @@ Useful heuristics during Google sync:
- cancelled instances should be handled as instance-level deletions
- payload ordering is not reliable enough to infer user intent by itself

This is why Compass sync logic keys off persisted state plus event properties instead of trying to reconstruct a single high-level Google UI action.
This is why Compass-to-Google event propagation keys off persisted state plus event properties instead of trying to reconstruct a single high-level Google UI action.

## Someday And Provider Semantics

Expand All @@ -114,7 +114,7 @@ Instead:

- `analyzeCompassTransition(...)` describes the implied Google effect
- `applyCompassPlan(...)` performs only Compass DB mutations
- `CompassSyncProcessor` executes Google create/update/delete after Compass persistence succeeds
- `CompassToGoogleEventPropagation` executes Google create/update/delete after Compass persistence succeeds

Delete-oriented Google effects should prefer the persisted DB `gEventId` when available, then fall back to the incoming payload `gEventId`.

Expand Down Expand Up @@ -143,7 +143,7 @@ Concrete examples from current tests:
- last persisted Compass event when a step returns one
- `googleDeleteEventId` resolved from persisted event first, otherwise planner fallback

The processor executes Google effects only after Compass persistence succeeds.
Compass-to-Google event propagation executes Google effects only after Compass persistence succeeds.

## Recurrence Sync Triage Runbook

Expand All @@ -155,7 +155,7 @@ Use this sequence when recurring edits behave unexpectedly:
3. Verify the planned `steps` order and `googleEffect` in unit tests:
- `compass.event.parser.test.ts`
- `compass.event.executor.test.ts`
- `compass.sync.processor.test.ts`
- `compass-to-google.event-propagation.test.ts`
4. Map each step to persistence calls in `executeStep(...)`:
- `create` -> `_createCompassEvent`
- `update` -> `_updateCompassEvent`
Expand Down Expand Up @@ -183,5 +183,5 @@ Good test anchors:

- `packages/backend/src/event/classes/compass.event.parser.test.ts`
- `packages/backend/src/event/classes/compass.event.executor.test.ts`
- `packages/backend/src/sync/services/sync/__tests__/compass.sync.processor.all-event.test.ts`
- `packages/backend/src/sync/services/sync/__tests__/compass-sync-processor-this-event/*.test.ts`
- `packages/backend/src/sync/services/event-propagation/__tests__/compass-to-google.all-event.test.ts`
- `packages/backend/src/sync/services/event-propagation/__tests__/compass-to-google-this-event/*.test.ts`
2 changes: 1 addition & 1 deletion packages/backend/src/__tests__/drivers/event.driver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { UserDriver } from "@backend/__tests__/drivers/user.driver";
import { mockGcalEvents } from "@backend/__tests__/mocks.gcal/factories/gcal.event.factory";
import calendarService from "@backend/calendar/services/calendar.service";
import mongoService from "@backend/common/services/mongo.service";
import { getGcalClient } from "@backend/sync/services/google-calendar-sync/google.calendar.client";
import { getGcalClient } from "@backend/sync/services/google-sync/gcal.client";

export class EventDriver {
static async generateV0Data(count = 3) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ import { type Schema_User } from "@core/types/user.types";
import dayjs from "@core/util/date/dayjs";
import { UserDriver } from "@backend/__tests__/drivers/user.driver";
import mongoService from "@backend/common/services/mongo.service";
import { getGcalClient } from "@backend/sync/services/google-calendar-sync/google.calendar.client";
import { getGcalClient } from "@backend/sync/services/google-sync/gcal.client";
import { updateSync } from "@backend/sync/services/records/sync-records.repository";
import { googleWatchService } from "@backend/sync/services/watch/google-watch.service";
import { updateSync } from "@backend/sync/util/sync.queries";

export class SyncDriver {
static async createSync(
export class GoogleSyncDriver {
static async createHealthyGoogleSync(
user: Pick<WithId<Schema_User>, "_id">,
defaultUser = false,
): Promise<void> {
Expand All @@ -31,7 +31,7 @@ export class SyncDriver {
);
}

static async generateV0Data(
static async generateLegacySyncWatchData(
numUsers = 3,
generateExpiredWatches = false,
): Promise<Array<WithId<Omit<Schema_Sync, "_id">>>> {
Expand Down
Loading