Skip to content

Add Gherkin behaviour tests with dummy runtime#1981

Closed
ifireball wants to merge 1 commit into
fullsend-ai:mainfrom
ifireball:cursor/5525b289
Closed

Add Gherkin behaviour tests with dummy runtime#1981
ifireball wants to merge 1 commit into
fullsend-ai:mainfrom
ifireball:cursor/5525b289

Conversation

@ifireball

Copy link
Copy Markdown
Contributor

Summary

  • Add defaults.runtime org config and --runtime install flag with shared runtime.ResolveFromConfig() selection in fullsend run.
  • Implement a dummy runtime that executes scripted sandbox operations and emits behaviour-results.json for deterministic assertions.
  • Introduce godog behaviour tests under e2e/behaviour/ with pluggable GitHub/GitHub Actions drivers, triage scenarios, CI job, and ADR/docs.

Test plan

  • go test ./...
  • go test -tags behaviour -c ./e2e/behaviour/...
  • go test -tags e2e -c ./e2e/admin/...
  • make behaviour-test against halfsend org pool with GITHUB_TOKEN and --runtime dummy orgs
  • CI behaviour job after adding E2E_BEHAVIOUR_GITHUB_TOKEN secret

Made with Cursor

Introduce defaults.runtime config, a scripted dummy runtime for deterministic
E2E validation, and godog-based behaviour tests against live GitHub/GHA.

Signed-off-by: Barak Korren <bkorren@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Jun 7, 2026

Copy link
Copy Markdown

Site preview

Preview: https://bddbbd25-site.fullsend-ai.workers.dev

Commit: 6a5393475a282d474bd1a161c7a8c22ad952747e

@ifireball

Copy link
Copy Markdown
Contributor Author

Closing to reopen from an upstream branch so CI can access repository secrets.

@fullsend-ai-review

Copy link
Copy Markdown

Review skipped — this PR is already closed.

The /fs-review command only reviews open pull requests.

Posted by fullsend post-review check

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #1981 — Wasted review agent run on short-lived draft PR

What happened: PR #1981 was opened as a draft from a fork at 10:40 UTC by a human contributor (ifireball) using Cursor. A review agent was dispatched 15 seconds later (run 27090168627). The author closed the PR at 10:47 UTC (7 minutes later) to reopen from an upstream branch so CI could access secrets. The review agent continued running until ~10:51 UTC, completing a full request-changes analysis with 8 findings — none of which were ever posted to the PR. PR #1982 was then opened at 10:49 UTC with identical content, triggering a second review run.

Waste: ~11 minutes of review agent compute (tokens + GHA minutes) were spent on a PR that was closed before the review finished. The same content was then re-reviewed on PR #1982.

Existing coverage: All improvement opportunities identified are already tracked by open issues:

  • #1715 — Skip review dispatch when PR is in draft state
  • #1870 — Skip review/fix dispatch when PR action is 'closed'
  • #1439 — Skip review dispatch when PR is already closed/merged
  • #1357 — Enable cancel-in-progress for review dispatches on the same PR

Recently closed issues #1566 and #887 suggest active work in this area.

No new proposals filed — existing issues comprehensively cover the improvements needed. This PR serves as a concrete real-world example validating the priority of those issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant