Flaky test: TestVerticalShardingFuzz — `or vector(...)` returns the vector() fallback instead of the LHS when vertical sharding is enabled

[CharlieTLe]

Summary

TestVerticalShardingFuzz (integration/query_fuzz_test.go:945) compares an unsharded Cortex (c1, res1) against an otherwise identical instance running -frontend.query-vertical-shard-size=2 (c2, res2). Both run the same build, so any difference is a vertical-sharding bug.

The fuzzer produced a query where the sharded instance returns the vector() fallback where PromQL requires the left-hand side of or. This is a genuine query correctness bug in vertical sharding, not a test artifact, and it is user-visible: <expr> or vector(0) is the idiomatic "default value" pattern in dashboards and alerting rules, and with vertical sharding enabled it can silently return 0 instead of the real value.

This is the same failure class as #5203 / #5204 / #5205 (absent, absent_over_time, scalar) and the same shape reported in #7547, which was closed as completed even though its fix PR #7551 is still open — master has no vector( handling in either isValidQuery or the sharding analyzer, so the bug was never actually addressed.

Why the sharded result is wrong

(
    max without (__name__, job, series) ({__name__="test_series_a",job="test"})
  or
    (-{__name__="test_series_b"} or vector(-0.3587905225767787))
)

• The left-hand side aggregates away every label the test_series_a series carry, so it yields the empty labelset {} with the max value 57.
• The right-hand side yields the three name-dropped test_series_b series plus, from vector(...), another {}.
• PromQL or returns all left-hand-side series, and from the right-hand side only series whose labelset is absent on the left. {} is present on the left, so the vector() sample must be dropped and the result for {} must be 57.

The unsharded instance does that. The sharded instance returns -0.3587905225767787.

Root cause

Vertical sharding appends a shard matcher to every vector selector (pkg/querysharding/util.go:29, InjectShardingInfo), so each shard evaluates the whole query over only the series it owns, and the shard results are concatenated. That is correct only while every output series is derived from a selector — then the shard that owns the series is the only shard that can produce it, and the analyzer's job is just to pick sharding labels that preserve that invariant.

vector(s) breaks the invariant. It synthesises a series with an empty labelset from a scalar and has no selector behind it, so every shard produces it.

For the query above the Thanos analyzer (vendor/github.com/thanos-io/thanos/pkg/querysharding/analyzer.go:92) computes "shard without (__name__, job, series)" from the max without (...) and the ors. The test_series_a series carry no other labels, so all three hash to the same shard:

• In the shard that owns them, the left-hand side yields {} => 57, the vector() sample collides with it and is correctly dropped by or. This shard returns {} => 57.
• In the other shard the left-hand side is empty, so nothing collides and or lets the vector() sample through. This shard returns {} => -0.3587905225767787.
• Merging the two shard results yields two {} series at the same timestamp, and the wrong one wins.

The analyzer already refuses to shard absent, absent_over_time and scalar for the same underlying reason — their results depend on data an individual shard cannot see — but vector is missing from that list:

case "absent_over_time", "absent", "scalar":
    isShardable = false
    return errNotShardable

Cortex's own wrapper NewDisableBinaryExpressionAnalyzer (pkg/querysharding/util.go:85) does not cover it either, and is in any case only installed when -querier.enable-parquet-queryable is set (pkg/cortex/modules.go:552).

This is a long-standing bug rather than a recent regression: vertical sharding was introduced in ee22f67 (#4863) and pkg/querysharding/ has not been touched in this area since. It becomes visible in CI only when the fuzzer happens to generate or vector(...) over data whose sharding labels collapse, which depends on the run's fuzz seed (newFuzzRand, #7552).

Reproduction

From the fuzzer, deterministically

$ CORTEX_IMAGE=<local build of master> FUZZ_SEED=1787336350 go test -v \
    -tags "integration,requires_docker,integration_query_fuzz" -timeout 2400s \
    -count=1 ./integration/ -run '^TestVerticalShardingFuzz$'

    query_fuzz_test.go:2246: case 92 results mismatch.
        instant query: (
            max without (__name__, job, series) ({__name__="test_series_a",job="test"})
          or
            (-{__name__="test_series_b"} or vector(-0.3587905225767787))
        )
        res1 len: 4 data: {} => 57 @[1787336921.48]
        ...
        res2 len: 4 data: {} => -0.3587905225767787 @[1787336921.48]
        ...
--- FAIL: TestVerticalShardingFuzz (11.81s)

Without the fuzzer

Pushing the same series by hand and issuing the query directly against an unsharded and a -frontend.query-vertical-shard-size=2 instance reproduces it, including in reduced form with a single or:

query: max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or vector(-0.3587905225767787)
  unsharded: {} => 57
  sharded  : {} => -0.3587905225767787

query: (max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or (-{__name__="test_series_b"} or vector(-0.3587905225767787)))
  unsharded: {} => 57
  sharded  : {} => -0.3587905225767787

Most recent occurrence

• Run: https://github.com/cortexproject/cortex/actions/runs/32510904119/job/96866421869 (attempt 2)
• Date: 2026-08-21
• Arch: ubuntu-24.04, amd64, build tag integration_query_fuzz
• Branch: release-1.22-pr-f-deprecate-max-exemplars (PR Formally deprecate -blocks-storage.tsdb.max-exemplars #7793), whose only change is a CLI help string — unrelated to this path.
• Seed: 1787336350
• Attempt 1 of the same run failed on TestExpandedPostingsCacheFuzz instead, for an unrelated cross-Prometheus-version reason tracked in Flaky test: TestExpandedPostingsCacheFuzz — Prometheus 3.9 merges same-labelset series after __name__ removal, latest release image errors #7803.

Failure excerpt

    query_fuzz_test.go:2246: case 92 results mismatch.
        instant query: (
            max without (__name__, job, series) ({__name__="test_series_a",job="test"})
          or
            (-{__name__="test_series_b"} or vector(-0.3587905225767787))
        )
        res1 len: 4 data: {} => 57 @[1787336290.092]
        {job="test", series="3", status_code="200"} => -77 @[1787336290.092]
        {job="test", series="4", status_code="400"} => -97 @[1787336290.092]
        {job="test", series="5", status_code="500"} => -117 @[1787336290.092]
        res2 len: 4 data: {} => -0.3587905225767787 @[1787336290.092]
        {job="test", series="3", status_code="200"} => -77 @[1787336290.092]
        {job="test", series="4", status_code="400"} => -97 @[1787336290.092]
        {job="test", series="5", status_code="500"} => -117 @[1787336290.092]
    query_fuzz_test.go:2251:
        	Error Trace:	/__w/cortex/cortex/integration/query_fuzz_test.go:2251
        	            				/__w/cortex/cortex/integration/query_fuzz_test.go:1058
        	Error:      	finished query fuzzing tests
        	Test:       	TestVerticalShardingFuzz
        	Messages:   	1 test cases failed
--- FAIL: TestVerticalShardingFuzz (14.32s)

Proposed fix

Fix the product, not the fuzz corpus: mark queries that use vector() as non-shardable, the same treatment absent / absent_over_time / scalar already get.

The analyzer that owns that list is vendored Thanos, so the Cortex-side change is a small wrapper analyzer in pkg/querysharding/util.go (mirroring the existing disableBinaryExpressionAnalyzer), installed unconditionally in initQueryFrontendTripperware. Queries without vector() are unaffected, so this costs sharding only for queries that cannot be sharded correctly today.

Filtering vector( out of the fuzz corpus — the approach proposed in #7547 and implemented in the still-open #7551 — would hide a real, user-visible wrong-results bug and is deliberately not what is done here.

A follow-up upstream change adding vector to the Thanos analyzer's non-shardable function list would let the Cortex wrapper be dropped again.

Verification

With the wrapper in place the divergence is gone, both by hand:

query: max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or vector(-0.3587905225767787)
  unsharded: {} => 57
  sharded  : {} => 57

and from the fuzzer at the CI seed:

$ CORTEX_IMAGE=<local build of master+fix> FUZZ_SEED=1787336350 go test -v \
    -tags "integration,requires_docker,integration_query_fuzz" -timeout 2400s \
    -count=1 ./integration/ -run '^TestVerticalShardingFuzz$'
--- PASS: TestVerticalShardingFuzz (11.58s)