Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: CI
on:
pull_request:
push:
branches: [main, v2]
branches: [main, dev, v2]
workflow_dispatch:

permissions:
Expand Down Expand Up @@ -890,4 +890,4 @@ jobs:
run: bash scripts/promote-to-dockerhub.sh
env:
GHCR_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
DOCKERHUB_TAGS: ${{ steps.tag.outputs.name }} latest
DOCKERHUB_TAGS: ${{ steps.tag.outputs.name }}${{ github.ref_name == 'main' && ' latest' || '' }}
28 changes: 17 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ If you add a migration, register it in `packages/state-postgres/src/migrations/i

## Architecture at a Glance

Sources and destinations are isolated connectors that only depend on `protocol`.
Sources and destinations are isolated connectors that only depend on `protocol`
and approved shared utilities (`logger`, `openapi`, `util-postgres`).
The engine loads connectors (in-process or subprocess), pipes source output through
destination input, and manages state checkpoints. See [docs/architecture/packages.md](docs/architecture/packages.md)
for the full dependency graph.
Expand All @@ -37,23 +38,28 @@ for the full dependency graph.

| Package | Purpose | Depends on |
| ------------------------------------ | --------------------------------------------------------- | ---------------------------------------- |
| `packages/protocol` | Message types, Source/Destination interfaces, Zod schemas | `zod` only |
| `packages/openapi` | Stripe OpenAPI spec fetching and parsing | standalone |
| `packages/source-stripe` | Stripe API source connector | `protocol`, `openapi` |
| `packages/destination-postgres` | Postgres destination connector | `protocol`, `util-postgres` |
| `packages/destination-google-sheets` | Google Sheets destination connector | `protocol` |
| `packages/state-postgres` | Postgres state store + migrations | `util-postgres` |
| `packages/util-postgres` | Shared Postgres utilities (upsert, rate limiter) | standalone |
| `packages/ts-cli` | Generic TypeScript module CLI runner | standalone |
| `packages/protocol` | Message types, Source/Destination interfaces, Zod schemas | `zod`, `citty`, `ix` |
| `packages/openapi` | Stripe OpenAPI spec fetching and parsing | `zod` |
| `packages/logger` | Structured logging (pino) + progress UI (ink) | `pino`, `ink`; peer: `protocol` |
| `packages/source-stripe` | Stripe API source connector | `protocol`, `openapi`, `logger` |
| `packages/destination-postgres` | Postgres destination connector | `protocol`, `util-postgres`, `logger` |
| `packages/destination-google-sheets` | Google Sheets destination connector | `protocol`, `logger` |
| `packages/state-postgres` | Postgres state store + migrations | `util-postgres`, `logger` |
| `packages/util-postgres` | Shared Postgres utilities (upsert, rate limiter) | `logger`, `pg` |
| `packages/hono-zod-openapi` | Hono + zod-openapi integration for spec generation | `hono`, `zod`, `zod-openapi` |
| `packages/test-utils` | Shared test helpers (servers, seeds, fixtures) | `destination-postgres`, `openapi`, `pg` |
| `packages/ts-cli` | Generic TypeScript module CLI runner | `citty` |
| `apps/engine` | Sync engine library + stateless CLI + HTTP API | `protocol`, connectors, `state-postgres` |
| `apps/service` | Stateful service (credentials, state management) | `engine` |
| `apps/service` | Pipeline management + Temporal workflows | `engine`, Temporal SDK |
| `apps/dashboard` | React web UI for pipeline management | `openapi-fetch`, `radix-ui` |
| `apps/visualizer` | Next.js data visualization tool | `next`, `source-stripe`, `pglite` |
| `apps/supabase` | Supabase edge functions (Deno runtime) | `protocol`, `engine`, connectors |
| `e2e/` | Cross-package conformance and layer tests | all packages |

## Key Rules

0. **This file is an index, not a rulebook** — before adding anything here, check if it belongs in [docs/architecture/principles.md](docs/architecture/principles.md), [docs/architecture/decisions.md](docs/architecture/decisions.md), or another doc first. Only add to AGENTS.md if no better home exists.
1. **Connector isolation** — sources never import destinations, both depend only on `protocol`. Enforced by `e2e/layers.test.ts`.
1. **Connector isolation** — sources never import destinations, both depend only on `protocol` + approved shared utilities. Enforced by `e2e/layers.test.ts`.
2. **State is a message** — connectors never access state storage directly. State in = `cursor_in`; state out = `SourceStateMessage`.
3. **Snake_case on the wire** — all Zod schemas and JSON wire format use snake_case.
4. **api_version is required** — always mandatory in Stripe source config. Never optional.
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Stripe Sync Engine

> [!WARNING]
> The `dev` branch is experimental and under active development and not yet documented.

For the original Supabase Stripe Sync Engine, see the [`og` branch](https://github.com/stripe/sync-engine/tree/og).

3 changes: 3 additions & 0 deletions apps/engine/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,14 @@
"@scalar/hono-api-reference": "^0.6",
"@stripe/sync-destination-google-sheets": "workspace:*",
"@stripe/sync-destination-postgres": "workspace:*",
"@stripe/sync-destination-sqlite": "workspace:*",
"@stripe/sync-hono-zod-openapi": "workspace:*",
"@stripe/sync-logger": "workspace:*",
"@stripe/sync-protocol": "workspace:*",
"@stripe/sync-source-metronome": "workspace:*",
"@stripe/sync-source-postgres": "workspace:*",
"@stripe/sync-source-stripe": "workspace:*",
"@stripe/sync-destination-redis": "workspace:*",
"@stripe/sync-ts-cli": "workspace:*",
"@stripe/sync-util-postgres": "workspace:*",
"citty": "^0.1.6",
Expand Down
51 changes: 48 additions & 3 deletions apps/engine/src/__generated__/openapi.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

126 changes: 122 additions & 4 deletions apps/engine/src/__generated__/openapi.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading