test(#3515): increase unit test timeout to 30s - #3576
fullsend-ai-coder[bot] wants to merge 1 commit into
Conversation
TestFilteringWithMixedRules in the conftest evaluator unit tests was panicking with a test timeout under CI load. The test performs I/O-heavy work (building tar.gz archives, compiling Rego policies via NewConftestEvaluatorWithNamespace) that occasionally exceeds the 10s budget, causing flaky failures such as those seen on PR #3295 (workflow runs 30586521570 and 30907557496). The failures are reproducibly transient: the same commit passes on re-run. Raise the -timeout flag for the unit test target in the Makefile from 10s to 30s so that inherently variable-latency operations (OPA policy compilation) have adequate headroom on shared CI runners. This matches the existing generative-test timeout and mirrors the alternative suggested by triage. Update AGENTS.md so the documented test-tag timeout table stays in sync with the Makefile. Note: pre-commit hooks were not run. The conforma/hooks pre-commit repo could not be fetched in the sandbox (git fetch returned HTTP 403); the sole configured hook (check-commit-message) validates commit messages rather than source files and does not apply to the Makefile/AGENTS.md diff. Closes #3515
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
Risk Assessment: moderate (2/5) DetailsA trivially scoped timeout bump (4 lines, bot-authored, exact issue match) kept at moderate rather than low by one protected-path touch (AGENTS.md) and the Makefile CI-workflow flag, with moderate historical churn on both files. |
ReviewFindingsMedium
|
fullsend-ai-review
Bot
added
the
requires-manual-review
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
/retest |
|
All PipelineRuns for this commit have already succeeded. Use |
This branch has not been deployed
1 participant
What:
Raises the
-timeoutflag on the unit-test invocation in theMakefilefrom10sto30s, and updates theTest Tagstable inAGENTS.mdso the documented unit-test timeout stays in sync.Makefile(unit test line):-timeout 10s→-timeout 30sAGENTS.md(Test Tags):unit(10s) →unit(30s)Why:
TestFilteringWithMixedRulesininternal/evaluator/conftest_evaluator_unit_core_test.gohas been intermittently panicking with a Go test timeout under CI load. The test does I/O-heavy work (creating temp directories, building tar.gz archives, and compiling Rego policies viaNewConftestEvaluatorWithNamespace()), and OPA capabilities-JSON parsing + policy compilation is inherently variable-latency. Under load on a shared CI runner it occasionally exceeds the previous 10s budget, panics, and fails the entire unit-test job.This behaviour is observable on Renovate PR #3295 (a security bump of
in-toto-golang), where theChecksworkflow failed 6 of 9 times partly due to this flake — with two failures (workflow runs 30586521570 and 30907557496) attributable toTestFilteringWithMixedRulestiming out and the same commit passing on re-run without any code change.30smatches the existing timeout for thegenerativetest tier and provides headroom for the OPA compilation code path without meaningfully lengthening a healthy run.Testing:
go test -race -tags=unit -timeout 30s -count=3 -run TestFilteringWithMixedRules ./internal/evaluator/— passes (ok github.com/conforma/cli/internal/evaluator 1.296s).make -n testrenders the intended command with-timeout 30s.Tickets:
Closes #3515
Post-script verification
agent/3515-increase-unit-test-timeout)65cf118af7ebe9efb68a3fd396d0eb249a001ca5..HEAD)