Flaky test: TestExpandedPostingsCacheFuzz — Prometheus 3.9 merges same-labelset series after __name__ removal, latest release image errors

[CharlieTLe]

Summary

TestExpandedPostingsCacheFuzz (integration/query_fuzz_test.go:427) is a cross-version compatibility test. It starts two single-binary Cortex instances:

• cortex-1 — the latest released image, resolved from the VERSION file by getLatestReleaseImage() (integration/util.go:41). At the time of writing VERSION is 1.21.1, so quay.io/cortexproject/cortex:v1.21.1, which vendors Prometheus v0.308.1 (3.8.1). Expanded postings cache disabled.
• cortex-2 — the current build ("" image ⇒ CORTEX_IMAGE), which vendors Prometheus v0.309.1 (3.9.1) on master. Expanded postings cache enabled.

(This corrects the description in #7545, which said cortex-1 runs v1.18.0 — the image is derived from VERSION and has moved on since.)

The fuzzer occasionally generates a query whose result contains two series that become identical after __name__ removal. Prometheus 3.9 changed the handling of exactly that case, so the old instance fails the query while the new one returns data. The test reports it as an error mismatch and fails.

This is a legitimate cross-Prometheus-version semantic difference. It is not an expanded-postings-cache bug — see the isolation experiment below.

Root cause

The failing query (formatted as the test logs it):

-(
    label_replace(
      rate({__name__="test_series_6"}[4m]),
      "__promqlsmith_dst_label__",
      "$1",
      "__name__",
      "(.*)"
    )
  or
    {__name__="test_series_6",test_label="test_label_value_2"}
)

1. rate(...) marks its output series DropName: true. With delayed name removal (the default since Prometheus 3.0) the __name__ label is not stripped immediately; it is stripped once, at the end of evaluation, in cleanupMetricLabels.

2. The or unions those name-dropped series with the raw selector's series, which still carry __name__="test_series_6". The enclosing unary - then marks every series in the result DropName: true.

3. At the end of evaluation cleanupMetricLabels removes __name__ from all of them, at which point the rate(...) series and the raw-selector series collapse onto the same labelset.

4. Prometheus ≤ 3.8 (vendor/.../promql/engine.go, pre-upgrade prometheus 3.9.1 #7535) failed the whole query at that point:

   if mat.ContainsSameLabelset() {
       ev.errorf("vector cannot contain metrics with the same labelset")
   }

   Prometheus 3.9 replaced that with mergeSeriesWithSameLabelset(mat), which merges the colliding series when their timestamps do not overlap and only errors when they do. Its doc comment describes this exact shape:

   mergeSeriesWithSameLabelset merges series in a matrix that have the same labelset after __name__ label removal. This happens when delayed name removal is enabled and operations like OR combine series that originally had different names but end up with the same labelset after dropping the name.

So cortex-1 (3.8.1) returns execution: vector cannot contain metrics with the same labelset and cortex-2 (3.9.1) returns no error. sameErrorClass (added in #7550) cannot reconcile this, because one side has no error at all.

The change landed in Cortex via 52a8537 — "upgrade prometheus 3.9.1" (#7535). git tag --contains 52a8537e2a is empty, i.e. no release carries it yet, so master and the newest released image are guaranteed to disagree on this shape until the next release. #7535 is not itself wrong — the test's version-skew filter simply does not know about this behaviour change.

isValidQuery(expr, skipBackwardIncompat=true) (integration/query_fuzz_test.go:2478) exists precisely to drop queries whose semantics changed across the embedded Prometheus versions (stddev, stdvar, quantile, predict_linear, atan2, …). It has no filter for this shape, so the query reaches the comparison.

The flakiness mechanism is the fuzz seed: newFuzzRand (#7552) seeds promqlsmith from the clock unless FUZZ_SEED is set, so only some seeds generate a colliding or.

Reproduction (deterministic)

The seed is logged, so the failure replays exactly:

$ CORTEX_IMAGE=<local build of master> FUZZ_SEED=1787335629 \
    go test -v -tags "integration,requires_docker,integration_query_fuzz" \
    -timeout 2400s -count=1 ./integration/ -run '^TestExpandedPostingsCacheFuzz$'

    query_fuzz_test.go:2270: integration fuzz random seed: overridden to 1787335629 via FUZZ_SEED
    query_fuzz_test.go:649: case 453 error mismatch.
        range query: -(
            label_replace(
              rate({__name__="test_series_6"}[4m]),
              "__promqlsmith_dst_label__",
              "$1",
              "__name__",
              "(.*)"
            )
          or
            {__name__="test_series_6",test_label="test_label_value_2"}
        )
        err1: execution: vector cannot contain metrics with the same labelset
        err2: <nil>
--- FAIL: TestExpandedPostingsCacheFuzz (9.65s)

Same case index (453), same query, same errors as CI.

The expanded postings cache is not the variable

The test changes two things at once (image and cache flag). Re-running the failing query with the two variables crossed over — latest release image with the cache enabled, vs. current build without it — shows the divergence follows the image:

stable-release-image (v1.21.1) + expanded postings cache ENABLED  -> err: execution: vector cannot contain metrics with the same labelset
HEAD                          + expanded postings cache DISABLED -> err: <nil>

Most recent occurrence

• Run: https://github.com/cortexproject/cortex/actions/runs/32510904119/job/96863335906 (attempt 1)
• Date: 2026-08-21
• Arch: ubuntu-24.04, amd64, build tag integration_query_fuzz
• Branch: release-1.22-pr-f-deprecate-max-exemplars (PR Formally deprecate -blocks-storage.tsdb.max-exemplars #7793), whose only change is a CLI help string — unrelated to this path.
• On attempt 2 of the same run TestExpandedPostingsCacheFuzz passed (different seed), and the same job failed on TestVerticalShardingFuzz instead — that is a separate, genuine product bug, tracked in Flaky test: TestVerticalShardingFuzz — or vector(...) returns the vector() fallback instead of the LHS when vertical sharding is enabled #7804.

Failure excerpt

    query_fuzz_test.go:2276: integration fuzz random seed: 1787335629 (override with FUZZ_SEED env var)
    query_fuzz_test.go:649: case 453 error mismatch.
        range query: -(
            label_replace(
              rate({__name__="test_series_6"}[4m]),
              "__promqlsmith_dst_label__",
              "$1",
              "__name__",
              "(.*)"
            )
          or
            {__name__="test_series_6",test_label="test_label_value_2"}
        )
        err1: execution: vector cannot contain metrics with the same labelset
        err2: <nil>
    query_fuzz_test.go:666:
        	Error Trace:	/__w/cortex/cortex/integration/query_fuzz_test.go:666
        	Error:      	finished query fuzzing tests
        	Test:       	TestExpandedPostingsCacheFuzz
        	Messages:   	1 test cases failed
--- FAIL: TestExpandedPostingsCacheFuzz (9.45s)

Proposed fix

Extend isValidQuery's skipBackwardIncompat branch to skip queries that use the or set operator, following the established pattern for quantile / atan2 / stddev (and the one used by #7549 / #7550 before it).

or is the operator that builds the offending result: it is the only one that can union series carrying different __name__s into a single result, which is the precondition for the post-name-removal collision. and and unless only ever return series taken from their left-hand side, so they are left alone.

Whether a particular or actually collides can only be determined by evaluating the query, so the filter is applied to the syntactic shape. skipBackwardIncompat=true is only passed by the cross-version tests, so or remains fully covered by the fuzz tests that compare two instances of the same build (TestVerticalShardingFuzz, TestProtobufCodecFuzz, TestParquetFuzz, …).

The filter is implemented as an AST walk for parser.LOR rather than a strings.Contains on the rendered query, so a label value containing or cannot accidentally drop a query.

Verification

With the filter in place, the previously failing seed passes:

$ CORTEX_IMAGE=<local build of master+fix> FUZZ_SEED=1787335629 go test -v \
    -tags "integration,requires_docker,integration_query_fuzz" -timeout 2400s \
    -count=1 ./integration/ -run '^TestExpandedPostingsCacheFuzz$'
    query_fuzz_test.go:2270: integration fuzz random seed: overridden to 1787335629 via FUZZ_SEED
--- PASS: TestExpandedPostingsCacheFuzz (22.54s)