Skip to content

Commit eaa53f7

Browse files
committed
Merge updated file foundation into Project files backend
2 parents 7a35c68 + 2325372 commit eaa53f7

379 files changed

Lines changed: 175579 additions & 13838 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/add-enrichment/SKILL.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Because enrichments run on Sim's hosted keys by default, **every provider tool y
2222

2323
## Architecture (what you're plugging into)
2424

25-
- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, projectFailure, mapOutput }` — `toolProvider` fills `projectFailure` with the standard HTTP projection; override it only for a provider whose failure shape is nonstandard (see `enrichments/provider-failures/`). Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe.
25+
- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, projectFailure, mapOutput, isVerified? }` — `toolProvider` fills `projectFailure` with the standard HTTP projection; override it only for a provider whose failure shape is nonstandard (see `enrichments/provider-failures/`). Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe.
2626
- **`enrichments/providers.ts`** — `toolProvider(...)` (typed passthrough) plus shared input helpers: `str(v)`, `normalizeDomain(v)`, `firstNonEmpty(arr)`, `splitName(fullName)`, and `projectEnrichmentProviderFailure`.
2727
- **`enrichments/run.ts`** — the server-only cascade runner. Calls `executeTool(provider.toolId, { ...params, _context: { workspaceId, userId } })`, accumulates hosted-key cost, returns the first non-empty mapped result. **You do not edit this** — it works for any registry entry.
2828
- **`enrichments/registry.ts`** — `ENRICHMENT_REGISTRY` / `ALL_ENRICHMENTS` / `getEnrichment`. Register new entries here.
@@ -37,7 +37,9 @@ For each output the enrichment produces, decide which existing tool provides it.
3737
- Its `params` accept what you can derive from table columns (read the tool's `params`).
3838
- Its `outputs` / `transformResponse` actually expose the field you need (read the real output shape — don't assume).
3939

40-
Order providers **cheapest / most-likely-to-hit first**; the cascade stops at the first non-empty result. Apollo and LinkedIn APIs are not hosted-safe (ToS) — never call them as providers.
40+
Order providers by **accuracy first, then price per hit**. Order decides which answer wins when two providers would disagree, so a cheaper but less accurate provider ahead of a better one trades quality for cost; reorder for price only with benchmark evidence. Among similarly accurate providers, put the ones that bill on a miss or charge a high price per hit after the cheaper, free-on-miss ones (check each vendor's billing docs and the tool's `hosting.pricing`); never move a more accurate provider behind a less accurate one to save cost. The cascade stops at the first non-empty, verified result. Apollo and LinkedIn APIs are not hosted-safe (ToS) — never call them as providers.
41+
42+
Prefer a vendor's own verified-only option (e.g. Prospeo `only_verified_email`) when it turns an unverified result into a free miss. Otherwise, when a provider can return an unconfirmed hit (a catch-all guess, a `risky` status), declare `isVerified(output)`: an unconfirmed hit is held while later providers look for a confirmed one, and fills the cell only if none does. `isVerified: false` marks a provider that never confirms its hits (a database record); it runs only while nothing is held.
4143

4244
## Step 2: Verify hosted-key support — chain to `/add-hosted-key` if missing
4345

@@ -136,7 +138,8 @@ export const ENRICHMENT_REGISTRY: EnrichmentRegistry = {
136138

137139
- [ ] Each output mapped to a real tool field (verified against the tool's `params`/`outputs`)
138140
- [ ] **Every provider tool has a `hosting` block — ran `/add-hosted-key` for any that didn't**
139-
- [ ] Providers ordered cheapest / most-likely-first; Apollo/LinkedIn not used
141+
- [ ] Providers ordered accuracy-first, then price per hit (billing on a miss checked against vendor docs); Apollo/LinkedIn not used
142+
- [ ] Providers that can return an unconfirmed hit declare `isVerified`
140143
- [ ] Enrichment file is client-safe (no `@/tools` import); uses `toolProvider` + shared helpers
141144
- [ ] `buildParams` returns `null` on insufficient inputs; `mapOutput` returns `null` on a miss
142145
- [ ] Registered in `enrichments/registry.ts`

‎.claude/rules/sim-testing.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ contracts, and demonstrated regressions.
3333
| Suffix | Needs | Run with | In CI |
3434
|--------|-------|----------|-------|
3535
| `*.test.ts(x)` | nothing; global mocks from `vitest.setup.ts` | `bun run --cwd <workspace> test` | `test` jobs |
36-
| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), optionally Redis (`TEST_REDIS_URL`) | `bun run --cwd <workspace> test --mode integration` | `integration` jobs, by glob (sharded per provisioning path) |
36+
| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), Redis (`TEST_REDIS_URL`; optional locally, required in CI) | `bun run --cwd <workspace> test --mode integration` | `integration` jobs, by glob (sharded per provisioning path) |
3737
| `*.live.test.ts` | provider APIs, hosted sandboxes, local runtimes, or sibling checkouts | `bun run --cwd apps/sim test --mode live <file>` | never |
3838
| `apps/desktop/e2e/*.spec.ts` | the packaged Electron app | Playwright | desktop E2E workflow |
3939
| `apps/sim/scripts/test-*-e2e.ts` | a running app over HTTP | its `package.json` script when one exists (`bun run test:scim:e2e`; `test:workflow-version-compare:e2e` adds `--no-env-file`), else `bun scripts/test-<suite>-e2e.ts` from apps/sim | `e2e` jobs (`.github/scripts/http-e2e.sh`) |
@@ -48,8 +48,8 @@ contracts, and demonstrated regressions.
4848
those checks. Isolate with a unique schema or generated IDs, and clean up in `afterAll`.
4949
- Integration files run one at a time against one shared database. A new `*.integration.ts` is
5050
picked up by CI with no workflow change, and the run writes the JSON report configured in
51-
`vitest.shared.ts`, which CI uploads. Never add a passing suite to the quarantine list in
52-
`apps/sim/vitest.config.ts`.
51+
`vitest.shared.ts`, which CI uploads. Never exclude a failing suite from the run in
52+
`apps/sim/vitest.config.ts`: fix it or delete it.
5353
- `bun run test:integration` starts disposable Postgres and Redis containers, provisions the schema,
5454
and runs both workspaces; pass filenames to narrow the `apps/sim` run.
5555

@@ -169,6 +169,7 @@ The suite's wall time is bound by the single Vite server thread that serves ever
169169
3. Mock heavy graphs a test does not need and the setup does not already mock: `@/blocks`,
170170
`@/triggers/registry`, `@/tools/generated/*`.
171171
4. No real timers: `vi.useFakeTimers()`, `flushMicrotasks()`, or `flushMacrotask()`.
172+
`check:test-patterns` (`real-wait`) fails on a new wall-clock wait in a unit test.
172173
5. Absolute imports only.
173174

174175
### Running

‎.cursor/rules/sim-testing.mdc‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ contracts, and demonstrated regressions.
3131
| Suffix | Needs | Run with | In CI |
3232
|--------|-------|----------|-------|
3333
| `*.test.ts(x)` | nothing; global mocks from `vitest.setup.ts` | `bun run --cwd <workspace> test` | `test` jobs |
34-
| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), optionally Redis (`TEST_REDIS_URL`) | `bun run --cwd <workspace> test --mode integration` | `integration` jobs, by glob (sharded per provisioning path) |
34+
| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), Redis (`TEST_REDIS_URL`; optional locally, required in CI) | `bun run --cwd <workspace> test --mode integration` | `integration` jobs, by glob (sharded per provisioning path) |
3535
| `*.live.test.ts` | provider APIs, hosted sandboxes, local runtimes, or sibling checkouts | `bun run --cwd apps/sim test --mode live <file>` | never |
3636
| `apps/desktop/e2e/*.spec.ts` | the packaged Electron app | Playwright | desktop E2E workflow |
3737
| `apps/sim/scripts/test-*-e2e.ts` | a running app over HTTP | its `package.json` script when one exists (`bun run test:scim:e2e`; `test:workflow-version-compare:e2e` adds `--no-env-file`), else `bun scripts/test-<suite>-e2e.ts` from apps/sim | `e2e` jobs (`.github/scripts/http-e2e.sh`) |
@@ -46,8 +46,8 @@ contracts, and demonstrated regressions.
4646
those checks. Isolate with a unique schema or generated IDs, and clean up in `afterAll`.
4747
- Integration files run one at a time against one shared database. A new `*.integration.ts` is
4848
picked up by CI with no workflow change, and the run writes the JSON report configured in
49-
`vitest.shared.ts`, which CI uploads. Never add a passing suite to the quarantine list in
50-
`apps/sim/vitest.config.ts`.
49+
`vitest.shared.ts`, which CI uploads. Never exclude a failing suite from the run in
50+
`apps/sim/vitest.config.ts`: fix it or delete it.
5151
- `bun run test:integration` starts disposable Postgres and Redis containers, provisions the schema,
5252
and runs both workspaces; pass filenames to narrow the `apps/sim` run.
5353

@@ -167,6 +167,7 @@ The suite's wall time is bound by the single Vite server thread that serves ever
167167
3. Mock heavy graphs a test does not need and the setup does not already mock: `@/blocks`,
168168
`@/triggers/registry`, `@/tools/generated/*`.
169169
4. No real timers: `vi.useFakeTimers()`, `flushMicrotasks()`, or `flushMacrotask()`.
170+
`check:test-patterns` (`real-wait`) fails on a new wall-clock wait in a unit test.
170171
5. Absolute imports only.
171172

172173
### Running

‎.github/scripts/check-project-rollout.py‎

Lines changed: 0 additions & 100 deletions
This file was deleted.

‎.github/scripts/ci-changes.sh‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env bash
2+
# Prints which checks a pull request's changes can affect, as `<flag>=true|false` lines for
3+
# $GITHUB_OUTPUT. A flag is `false` only when every changed file is on an allowlist whose jobs
4+
# fully cover it. A file on no allowlist sets every flag but `desktop_live` (which keeps its own
5+
# skip list) to `true`; a change under .github/, an empty diff, or any failure to work out the
6+
# diff sets all of them.
7+
#
8+
# full integration, e2e, e2e-build, e2e-mobile: any file not allowlisted below
9+
# test the Vitest shards and test-types: content, docs app, and desktop changes,
10+
# plus `full`
11+
# build the app's `next build`: content changes, plus `full`
12+
# desktop_live the live Electron suite: any file outside its own skip list
13+
#
14+
# lint always runs and is not gated.
15+
#
16+
# Usage: ci-changes.sh <base-sha>
17+
set -u
18+
19+
# Posts, author profiles, and the public assets only posts reference. The test shards read them
20+
# (lib/content/og-image.test.ts) and `next build` prerenders every post with its MDX components,
21+
# which check:library-content compiles but never renders. Components under content/ are code.
22+
content='^apps/sim/content/([^/]+/)*[^/]+\.mdx?$|^apps/sim/content/authors/[^/]+\.json$|^apps/sim/public/(authors|blog|changelog|library)/'
23+
# Its own workspace tests, its type-check and docs-manifest audit in lint, and apps/sim tests
24+
# that read its pages (tools/netsuite/netsuite.test.ts).
25+
docs='^apps/docs/'
26+
# Nothing outside the Electron app imports it; the desktop bridge and protocol packages are
27+
# consumed by apps/sim and stay unlisted.
28+
desktop='^apps/desktop/'
29+
# Agent guidance, read only by the lint job's audits.
30+
guidance='^(\.agents|\.claude|\.cursor)/|^([^/]+/)*(AGENTS|CLAUDE)\.md$|^README\.md$'
31+
# The live desktop suite drives apps/desktop against the app, so it skips more than the rest.
32+
desktop_live_skip="$guidance"'|^(apps/docs/|apps/pii/|apps/sim/content/|apps/sim/public/(authors|blog|changelog|library)/|packages/(python-sdk|ts-sdk)/)|\.mdx?$|(^|/)LICENSE$'
33+
34+
emit() {
35+
printf 'full=%s\ntest=%s\nbuild=%s\ndesktop_live=%s\n' "$1" "$2" "$3" "$4"
36+
}
37+
run_all() {
38+
emit true true true true
39+
echo "Running every check: $1" >&2
40+
exit 0
41+
}
42+
matches() { printf '%s\n' "$names" | grep -qE "$1"; }
43+
outside() { printf '%s\n' "$names" | grep -qvE "$1"; }
44+
45+
base=${1:-}
46+
[ -n "$base" ] || run_all 'no base commit'
47+
git fetch --quiet --depth=1 origin "$base" || run_all "could not fetch $base"
48+
names=$(git diff --no-renames --name-only "$base" HEAD) || run_all "could not diff against $base"
49+
[ -n "$names" ] || run_all 'no changed files listed'
50+
if matches '^\.github/'; then run_all 'a CI file changed'; fi
51+
52+
desktop_live=false
53+
if outside "$desktop_live_skip"; then desktop_live=true; fi
54+
if outside "$content|$docs|$desktop|$guidance"; then
55+
emit true true true "$desktop_live"
56+
unlisted=$(printf '%s\n' "$names" | grep -vE "$content|$docs|$desktop|$guidance" | head -n 1)
57+
echo "Running every check (desktop_live=$desktop_live): $unlisted is not on an allowlist" >&2
58+
exit 0
59+
fi
60+
61+
test=false build=false
62+
if matches "$content|$docs|$desktop"; then test=true; fi
63+
if matches "$content"; then build=true; fi
64+
emit false "$test" "$build" "$desktop_live"
65+
echo "Scoped checks: test=$test build=$build desktop_live=$desktop_live" >&2

‎.github/scripts/ci-gate.sh‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/usr/bin/env bash
2+
# The `ci` job's verdict: the single required check, so it must never pass a run whose checks did
3+
# not all pass. Reads the job's `needs` context (`toJSON(needs)`) from $NEEDS and exits non-zero,
4+
# naming each offending job, unless every job succeeded or was skipped as `changes` intended.
5+
#
6+
# A skip passes only for a gated job whose `changes` flag is exactly `false`:
7+
#
8+
# integration, e2e, e2e-build, e2e-mobile full
9+
# test, test-types test
10+
# build build
11+
# desktop-live desktop_live
12+
#
13+
# Every other skip fails: an ungated job (lint, `changes` itself), a gated job whose flag is
14+
# `true`, missing, or anything else (a failed or skipped `changes` leaves no outputs). A failure
15+
# or a cancellation always fails.
16+
#
17+
# Usage: NEEDS='<toJSON(needs)>' ci-gate.sh
18+
set -euo pipefail
19+
20+
needs=${NEEDS:-}
21+
if ! jq -e 'type == "object"' <<< "$needs" > /dev/null 2>&1; then
22+
echo "::error::The needs context is not a JSON object: '$needs'"
23+
exit 1
24+
fi
25+
26+
failed="$(jq -r '
27+
{"integration": "full", "e2e": "full", "e2e-build": "full", "e2e-mobile": "full", "test": "test",
28+
"test-types": "test", "build": "build", "desktop-live": "desktop_live"} as $gate
29+
| (.changes.outputs // {}) as $flags
30+
| to_entries[]
31+
| select(.value.result != "success")
32+
| select(.value.result != "skipped" or $gate[.key] == null or $flags[$gate[.key]] != "false")
33+
| "\(.key): \(.value.result)"
34+
' <<< "$needs")"
35+
if [ -n "$failed" ]; then
36+
echo "::error::Checks did not pass:"
37+
echo "$failed"
38+
exit 1
39+
fi
40+
echo "All checks passed."

0 commit comments

Comments
 (0)