Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/bright-replies-copy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@chatcops/widget": minor
---

Add a localized copy action to completed assistant replies.
5 changes: 5 additions & 0 deletions .changeset/cli-simulated-visitor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@chatcops/cli": minor
---

A test in an eval file can use `simulate` instead of `turns`: a second model (the judge) plays the visitor from a `persona` for up to `maxTurns` messages, stopping when it answers `DONE` or, with `stopWhen: lead-captured`, when the bot captures a lead. A test-level `expect` is checked once, on the last turn, with the whole conversation. The visitor's calls are recorded and replayed like the judge's, and count in the cost and `--max-cost`.
6 changes: 6 additions & 0 deletions .changeset/silver-webhooks-retry-limits.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@chatcops/core": minor
"@chatcops/server": minor
---

Allow webhook configs to set per-endpoint delivery timeouts and maximum attempts.
5 changes: 5 additions & 0 deletions .changeset/theme-css-tokens.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@chatcops/widget": minor
---

Add typed CSS token overrides to `theme` and support changing them with `update()`.
5 changes: 5 additions & 0 deletions .changeset/theme-token-surface-raised.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@chatcops/widget': patch
---

`theme.tokens` takes `surface-raised`, the background of the header menu.
95 changes: 95 additions & 0 deletions .github/workflows/accepted-label.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
name: Accepted label

# A merged PR counts on the leaderboard (oss.codercops.com) only with the
# codercops-accepted label, and only for an issue that was assigned to its
# author (CONTRIBUTING.md). PRs merged in October also get hacktoberfest-accepted.
# Maintainers add the label when they merge. This adds it when that was missed,
# by the same rule as the claim check: every issue the PR closes is assigned to
# the author. It runs once, at the merge, so removing the label later sticks.
#
# It uses pull_request_target so it can label fork PRs. That is safe here: the
# job never checks out or runs the PR's code.
on:
pull_request_target:
branches: [dev]
types: [closed]

permissions:
issues: read
pull-requests: write

jobs:
label:
name: Label a merged contributor PR
# Maintainers and bots don't claim issues, and spam never counts. Outside
# collaborators (contributors with Triage) are labeled like everyone else.
if: >-
github.event.pull_request.merged == true &&
github.event.pull_request.user.type != 'Bot' &&
!contains(fromJSON('["OWNER", "MEMBER"]'), github.event.pull_request.author_association) &&
!contains(github.event.pull_request.labels.*.name, 'codercops-accepted') &&
!contains(github.event.pull_request.labels.*.name, 'spam')
runs-on: ubuntu-latest
steps:
- name: Add the label if every linked issue is assigned to the author
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
PR: ${{ github.event.pull_request.number }}
MERGED_AT: ${{ github.event.pull_request.merged_at }}
run: |
set -euo pipefail

pr=$(gh api graphql \
-f owner="${REPO%/*}" -f name="${REPO#*/}" -F pr="$PR" \
-f query='query($owner: String!, $name: String!, $pr: Int!) {
repository(owner: $owner, name: $name) {
pullRequest(number: $pr) {
author { login }
body
closingIssuesReferences(first: 20) {
nodes { number repository { nameWithOwner } assignees(first: 20) { nodes { login } } }
}
}
}
}' \
--jq '.data.repository.pullRequest')
author=$(jq -r '.author.login // ""' <<<"$pr")

# The issues this PR closed ("Closes #12" in the description).
linked=$(jq --arg repo "$REPO" '[ .closingIssuesReferences.nodes[] | {
ref: (if .repository.nameWithOwner == $repo then "#\(.number)" else "\(.repository.nameWithOwner)#\(.number)" end),
assignees: [.assignees.nodes[].login]
} ]' <<<"$pr")

# Same fallback as the claim check: read "Closes #12" from the description.
if [ "$(jq length <<<"$linked")" -eq 0 ]; then
numbers=$(jq -r '[ (.body // "") | scan("(?i)\\b(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\\s*:?\\s+#([0-9]+)") | .[0] ] | unique | .[]' <<<"$pr")
for number in $numbers; do
if issue=$(gh api "repos/$REPO/issues/$number" --jq 'select(.pull_request == null) | {ref: "#\(.number)", assignees: [.assignees[].login]}' 2>/dev/null) && [ -n "$issue" ]; then
linked=$(jq --argjson issue "$issue" '. + [$issue]' <<<"$linked")
fi
done
fi

if [ "$(jq length <<<"$linked")" -eq 0 ]; then
echo "::notice title=Not labeled::This PR isn't linked to an issue, so it doesn't get codercops-accepted."
exit 0
fi

missing=$(jq -r --arg author "$author" '
[ .[]
| select(all(.assignees[]; ascii_downcase != ($author | ascii_downcase)))
| .ref ]
| join(", ")' <<<"$linked")

if [ -n "$missing" ]; then
echo "::notice title=Not labeled::$missing isn't assigned to @$author, so this PR doesn't get codercops-accepted."
exit 0
fi

labels=(-f "labels[]=codercops-accepted")
# Hacktoberfest only counts PRs merged in October (UTC).
if [ "${MERGED_AT:5:2}" = "10" ]; then labels+=(-f "labels[]=hacktoberfest-accepted"); fi
gh api -X POST "repos/$REPO/issues/$PR/labels" "${labels[@]}" --silent
echo "Labeled: $(jq -r '[.[].ref] | join(", ")' <<<"$linked") assigned to @$author"
2 changes: 1 addition & 1 deletion .github/workflows/claim-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
# Maintainers and bots don't claim issues.
if: >-
github.event.pull_request.user.type != 'Bot' &&
!contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.pull_request.author_association)
!contains(fromJSON('["OWNER", "MEMBER"]'), github.event.pull_request.author_association)
runs-on: ubuntu-latest
steps:
- name: Check the linked issue
Expand Down
2 changes: 1 addition & 1 deletion .size-limit.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
{
"name": "widget, single file (chatcops.full.min.js)",
"path": "packages/widget/dist/chatcops.full.min.js",
"limit": "22 KB",
"limit": "23 KB",
"gzip": true
},
{
Expand Down
1 change: 1 addition & 0 deletions .well-known/funding-manifest-urls
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
https://oss.codercops.com/funding.json
18 changes: 9 additions & 9 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@

Thanks for helping out. This guide covers setup, how we work, and what a good PR looks like.

## Hacktoberfest 2026
## How to contribute

We're taking part in Hacktoberfest this year, and first-time contributors are very welcome.
First-time contributors are very welcome.

1. Pick an open issue labeled [`hacktoberfest`](https://github.com/codercops/chatcops/issues?q=is%3Aissue+is%3Aopen+label%3Ahacktoberfest+no%3Aassignee) or [`good first issue`](https://github.com/codercops/chatcops/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22+no%3Aassignee) that nobody is assigned to.
1. Pick an open issue labeled [`good first issue`](https://github.com/codercops/chatcops/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22+no%3Aassignee) that nobody is assigned to.
2. Comment on it to ask for it. I'll assign it to you, usually within a day. Please don't start on an issue that's assigned to someone else.
3. Take one issue at a time. Once your PR is merged, grab the next one. After 3 merged PRs with us you can hold two issues at once.
4. If you're assigned and there's no PR or update for 5 days, I'll free the issue up for someone else. Just comment if you need more time.
5. Open your PR against `dev` here in codercops/chatcops and put `Closes #<issue number>` in the description. For an issue that stays open across several PRs, like one language of the translations issue, write `Part of #<issue number>` instead. A PR merged inside your own fork never reaches us, so it doesn't count. A check on the PR ("Claim check") fails if that issue isn't assigned to you. Another one ("Description check") fails while the "How I tested this" or "AI assistance" section of the description is missing or empty.

PRs count for Hacktoberfest once they're merged or labeled `hacktoberfest-accepted`. PRs that aren't linked to an issue, only reformat code, or change things nobody asked for will be closed, and spammy ones get the `spam` label. If you have an idea that isn't an issue yet, open an issue first so we can agree on it before you write code. The full rules are under [Ground rules](#ground-rules) below.
A merged PR counts on the monthly leaderboard once it's labeled `codercops-accepted`. A workflow adds that label when the PR is merged, as long as every issue it closes is assigned to you. PRs that aren't linked to an issue, only reformat code, or change things nobody asked for will be closed, and spammy ones get the `spam` label. If you have an idea that isn't an issue yet, open an issue first so we can agree on it before you write code. The full rules are under [Ground rules](#ground-rules) below.

Stuck? Ask on the issue, or in `#oss-help` on our [Discord](https://discord.gg/mpGdJyjaRQ). Questions are welcome. Claims still happen on the issue, so the assignment stays visible to everyone.

Expand All @@ -38,7 +38,7 @@ These get closed and labeled `spam`:
- claiming a pile of issues at once, or bulk and automated submissions
- untested generated code

Two spam PRs and you won't be assigned more issues this month, and you're out of the CODERCOPS goodies.
Two spam PRs and you won't be assigned more issues this month, and your PRs stop counting on the leaderboard.

### Reviews

Expand All @@ -48,15 +48,15 @@ For a bug fix, check that your new test fails without the fix: undo the change i

I review most PRs within a few days. Please don't ping repeatedly or DM maintainers about a review. If three days pass with no reply, one follow-up comment on the PR is fine.

### Goodies
### Monthly leaderboard

The top 5 contributors this October get a CODERCOPS T-shirt and stickers. You need at least 5 merged PRs labeled `hacktoberfest-accepted` across chatcops, ogcops, toolbelt and discord-bot, merged between Oct 1 and Oct 31 (UTC), and ties go to whoever got there first. To keep it fair:
Every month has its own leaderboard at [oss.codercops.com/leaderboard](https://oss.codercops.com/leaderboard). It counts PRs labeled `codercops-accepted` across chatcops, ogcops, toolbelt and discord-bot that were merged in that month (UTC). You're ranked from 3 merged PRs, and the top 5 get a shout-out on our X and LinkedIn pages. To keep it fair:

- only PRs for issues that were assigned to you count
- a fix split into several PRs counts once
- what counts is the maintainers' call, and that call is final

These goodies come from CODERCOPS through our own open source program. They're separate from Hacktoberfest, so anything Hacktoberfest itself gives out is a different thing with its own rules.
People who keep contributing can be invited as maintainers with Triage access. [GOVERNANCE.md](https://github.com/codercops/.github/blob/main/GOVERNANCE.md) explains how that works and who maintains what.

## Prerequisites

Expand Down Expand Up @@ -210,7 +210,7 @@ ANTHROPIC_API_KEY=sk-... npx tsx src/examples/express-server.ts

`npx` downloads `tsx` the first time.

The widget has a size budget, gzipped: 5 kB for the script tag loader (`dist/chatcops.min.js`), 19 kB for the chat panel it loads when the chat opens (`dist/chatcops.panel.js`), 2 kB for the pre-chat form chunk, 22 kB for the single-file build (`dist/chatcops.full.min.js`) and 32 kB for the unminified ESM build. CI checks them with `pnpm size` after a build (limits in `.size-limit.json`). To save bytes, the minifier renames the private class members listed in `packages/widget/mangle.ts`. A name can go on that list only if the code touches it through `this` and nowhere else, which `tests/mangle.test.ts` checks. A config field, a request body key or a DOM property renamed there would break only the CDN build.
The widget has a size budget, gzipped: 5 kB for the script tag loader (`dist/chatcops.min.js`), 19 kB for the chat panel it loads when the chat opens (`dist/chatcops.panel.js`), 2 kB for the pre-chat form chunk, 23 kB for the single-file build (`dist/chatcops.full.min.js`) and 32 kB for the unminified ESM build. CI checks them with `pnpm size` after a build (limits in `.size-limit.json`). To save bytes, the minifier renames the private class members listed in `packages/widget/mangle.ts`. A name can go on that list only if the code touches it through `this` and nowhere else, which `tests/mangle.test.ts` checks. A config field, a request body key or a DOM property renamed there would break only the CDN build.

### Browser tests

Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,10 @@ Questions, or want to chat with other contributors? Join the [CODERCOPS Discord]
<img src="https://contrib.rocks/image?repo=codercops/chatcops" />
</a>

## Maintainers and funding

ChatCops is maintained by CODERCOPS together with contributors who have Triage access. [GOVERNANCE.md](https://github.com/codercops/.github/blob/main/GOVERNANCE.md) lists who they are and how to become one. The project is free and MIT licensed, and CODERCOPS doesn't sell it or a hosted version of it. If it's useful to you, you can [sponsor the maintainers](https://github.com/sponsors/anurag629). The money pays for review time and hosting.

## License

MIT
18 changes: 16 additions & 2 deletions api/cli.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ export declare function findFixtures(cwd: string, patterns: string[]): Promise<s
export interface Fixture {
version: 1;
name: string;
defaults?: Partial<Omit<TestCase, 'name' | 'turns'>>;
defaults?: Partial<Omit<TestCase, 'name' | 'turns' | 'simulate' | 'expect'>>;
tests: TestCase[];
}

Expand Down Expand Up @@ -114,7 +114,9 @@ export interface TestCase {
mocks?: {
tools?: Record<string, ToolResult>;
};
turns: Turn[];
turns?: Turn[];
simulate?: Simulate;
expect?: Assertion[];
repeat?: number;
passRate?: number;
}
Expand Down Expand Up @@ -168,6 +170,8 @@ export interface TurnResult {
error?: string;
model?: string;
costUsd?: number;
simulated?: boolean;
visitorCostUsd?: number;
}
```

Expand Down Expand Up @@ -216,6 +220,10 @@ interface Judge {
reason: string;
score: number;
}>;
play(persona: string, conversation: TurnResult[]): Promise<{
costUsd?: number;
message: string;
}>;
}

interface PageContext {
Expand All @@ -239,6 +247,12 @@ interface RunSummary {
costUsd?: number;
}

interface Simulate {
persona: string;
maxTurns: number;
stopWhen?: 'lead-captured';
}

declare class Spend {
readonly max: number | undefined;
spent: number;
Expand Down
2 changes: 2 additions & 0 deletions api/core.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -999,6 +999,8 @@ export interface VoyageRerankerOptions {
export interface WebhookConfig {
url: string;
events: string[];
timeoutMs?: number;
maxAttempts?: number;
secret?: string;
allowPrivateNetwork?: boolean;
}
Expand Down
38 changes: 38 additions & 0 deletions api/widget.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ export interface ThemeConfig {
bgColor?: string;
fontFamily?: string;
borderRadius?: string | number;
tokens?: WidgetTokens;
position?: 'bottom-right' | 'bottom-left';
}

Expand Down Expand Up @@ -213,6 +214,9 @@ export interface WidgetLocaleStrings {
retryButton: string;
messageWaiting: string;
regenerateButton: string;
copyButton: string;
copySuccess: string;
copyFailure: string;
suggestionsLabel: string;
unreadCount: string;
sourcesLabel: string;
Expand All @@ -225,6 +229,10 @@ export interface WidgetLocaleStrings {
privacy: string;
}

export type WidgetTokenName = (typeof TOKEN_NAMES)[number];

export type WidgetTokens = Partial<Record<WidgetTokenName, string>>;

export type WidgetUpdate = Pick<WidgetConfig, 'branding' | 'welcomeMessage' | 'disclosure' | 'suggestions' | 'placeholder' | 'strings' | 'privacyUrl'> & {
theme?: Omit<ThemeConfig, 'position'>;
};
Expand All @@ -238,3 +246,33 @@ The same for each of: `ar`, `de`, `es`, `fr`, `he`, `hi`, `it`, `ja`, `ko`, `nl`
declare const _default: Partial<WidgetLocaleStrings>;
export { _default as default };
```

## Types no entry exports

They're part of the API above all the same: a change to one is a change to the API.

```ts
declare const TOKEN_NAMES: readonly [
"accent",
"accent-hover",
"on-accent",
"accent-contrast",
"text",
"bg",
"bg-surface",
"surface-raised",
"bg-input",
"border",
"text-secondary",
"danger",
"code-bg",
"pre-bg",
"shadow",
"font",
"radius",
"fab-size",
"panel-width",
"panel-height",
"focus-ring"
];
```
11 changes: 6 additions & 5 deletions packages/cli/src/test/command.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { EXIT, wholeNumber, type Io, type Values } from '../main.js';
import { assertionName, unknownAssertions } from './assertions/index.js';
import { Spend } from './cost.js';
import { resolveJudge } from './judge.js';
import { DEFAULT_PATTERN, findFixtures, loadFixture, matchesFilter } from './load.js';
import { DEFAULT_PATTERN, assertionsOf, findFixtures, loadFixture, matchesFilter } from './load.js';
import type { CassetteMode, Cassettes } from './cassette.js';
import { runFixtures, withMocks, withoutKeys, type FixtureJob } from './run.js';
import { REPORTERS } from './reporters/index.js';
Expand All @@ -33,7 +33,7 @@ export async function runTests(
const fixtures: FixtureJob[] = [];
for (const file of files) {
const fixture = await loadFixture(file);
const unknown = unknownAssertions(fixture.tests.flatMap((test) => test.turns.flatMap((turn) => turn.expect ?? [])));
const unknown = unknownAssertions(fixture.tests.flatMap(assertionsOf));
if (unknown.length) throw new ConfigError(`${relative(io.cwd, file)} uses assertions that don't exist: ${unknown.join(', ')}`);
for (const test of fixture.tests) {
try {
Expand Down Expand Up @@ -63,7 +63,8 @@ export async function runTests(
throw new ConfigError(`The server config's provider can't start: ${(error as Error).message}`);
}

const usesJudge = fixtures.some((job) => job.tests.some((test) => test.turns.some((turn) => turn.expect?.some((a) => assertionName(a).name === 'judge'))));
// A simulated visitor is played by the judge model too.
const usesJudge = fixtures.some((job) => job.tests.some((test) => test.simulate || assertionsOf(test).some((a) => assertionName(a).name === 'judge')));
const judge = usesJudge
? await resolveJudge({
...(cli.judge ? { configured: cli.judge } : {}),
Expand Down Expand Up @@ -106,13 +107,13 @@ export async function runTests(
},
});
const elapsedMs = performance.now() - started;
const costs = results.flatMap((r) => r.runs.flatMap((run) => run.turns.flatMap(({ turn, results: checks }) => [turn.costUsd, ...checks.map((c) => c.costUsd)])));
const costs = results.flatMap((r) => r.runs.flatMap((run) => run.turns.flatMap(({ turn, results: checks }) => [turn.costUsd, turn.visitorCostUsd, ...checks.map((c) => c.costUsd)])));
const priced = costs.filter((cost): cost is number => cost !== undefined);
const costIncomplete =
results.some((result) =>
result.runs.some(
(run) =>
run.turns.some(({ turn }) => turn.costUsd === undefined) ||
run.turns.some(({ turn }) => turn.costUsd === undefined || (turn.simulated && turn.visitorCostUsd === undefined)) ||
run.turns.some(({ results: checks }) => checks.some((check) => assertionName(check.assertion).name === 'judge' && check.costUsd === undefined))
)
);
Expand Down
Loading