Skip to content

Fix flaky acts_as_model specs caused by RubyLLM.logger spec leaving config nil#815

Open
cgmoore120 wants to merge 3 commits into
crmne:mainfrom
cgmoore120:fix/logger-spec-config-pollution
Open

Fix flaky acts_as_model specs caused by RubyLLM.logger spec leaving config nil#815
cgmoore120 wants to merge 3 commits into
crmne:mainfrom
cgmoore120:fix/logger-spec-config-pollution

Conversation

@cgmoore120

Copy link
Copy Markdown
Contributor

Problem

The acts_as_model "model registry integration" specs (loads models from database when configured, finds models from database) fail intermittently under the parallel test queue (bin/rspec-queue), with errors like:

Failure/Error: expect(models.all.map(&:id)).to include('test-model')

The failures are order-dependent: they only appear when certain specs land in the same worker, so they surface non-deterministically depending on how test-queue distributes examples (more likely on slower Ruby versions).

Root cause

The RubyLLM.logger spec resets the global config/logger in its after hook by setting them to nil instead of restoring the originals:

after do
  described_class.instance_variable_set(:@config, nil)
  described_class.instance_variable_set(:@logger, nil)
end

Once @config is left nil, the next access lazily rebuilds a fresh Configuration. But config.model_registry_source is only ever set once — in RubyLLM::ActiveRecord::ActsAs.included at load time — so the rebuilt config no longer has it. Any later spec in the same worker that relies on the database-backed model registry (Models#load_models reads config.model_registry_source) then silently falls back to the JSON registry and fails to find the test records.

This reproduces deterministically by running the two specs in order:

rspec spec/ruby_llm_spec.rb spec/ruby_llm/active_record/acts_as_model_spec.rb --order defined
# => acts_as_model registry integration: 2 failures

Fix

Use an around hook that saves and restores the original @config/@logger, so the logger spec leaves global state untouched. After the fix the same ordered run passes (22 examples, 0 failures), and ruby_llm_spec itself stays green.

The `.logger` spec reset `RubyLLM.@config`/`@logger` to nil in its
`after` hook instead of restoring the originals. Because the AR
integration sets `config.model_registry_source` only once (in
ActsAs.included at load time), a nil-ed config is rebuilt without that
source, so any later spec in the same parallel test-queue worker that
relies on the database-backed model registry (acts_as_model registry
integration) silently falls back to the JSON registry and fails.

Use an around hook that saves and restores the original config/logger so
the global state is left untouched. This is order-dependent and only
surfaced intermittently under the parallel queue.
@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.45%. Comparing base (44f3cc0) to head (42bbd3f).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #815      +/-   ##
==========================================
- Coverage   81.47%   81.45%   -0.03%     
==========================================
  Files         165      165              
  Lines        7402     7402              
  Branches     1233     1233              
==========================================
- Hits         6031     6029       -2     
- Misses        874      875       +1     
- Partials      497      498       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant