A cached matrix measures nothing and reports success - #161
Merged
Merged
Conversation
The version-matrix job has not run the matrix since the commit that first produced the table. Gradle restored `:kdrant-transport-rest:jvmTest` FROM-CACHE on every run after it, so the test never executed, wrote no report, uploaded no artifact, and the job reported success the whole time. `if-no-files-found: ignore` swallowed the missing upload. The README's table is the evidence: it names Qdrant v1.19.0 and 39 contract cases while the pin is v1.19.1 and the contract has 46. It is three releases stale and nothing went red. So the task runs with `--rerun`, because this job's product is a measurement rather than a verdict and Gradle cannot tell the difference, and a missing table is now an upload error. A cell that fails against an older server is what this job exists to publish and still must not fail it; a run that measured nothing is a defect rather than a result, and those two were being treated the same way. Also `workflow_dispatch`, so the matrix can be re-measured without pushing a commit. Qdrant releasing a minor is the usual reason to want that.
|
This pull request adds no capability the code did not already have. 3 files read · 2157 ms · against |
The table now comes from run 34693421161, which is the first time the matrix has actually executed since the commit that produced the old one. It does not read all-green, and that is the table finally doing its job: v1.19.1 46/46 v1.18.3 42/46 the four 1.19 features v1.17.1 41/46 plus TurboQuant v1.16.3 40/46 plus relevance feedback Each failing cell names a feature the server does not have, so the answer to "we are on 1.17, can we use this" is now answerable from the README instead of being a question. The prose says which version each missing feature needs, and the numbers were checked against Qdrant's own release notes rather than inferred from the cells: TurboQuant is 1.18.0, relevance feedback is 1.17.0. That second one corrects the 2.3.0 section. Its headline listed relevance feedback as part of Qdrant 1.19's surface and it is not: Qdrant has served it since 1.17.0, and what Kdrant added was the DSL for it. The matrix caught that on its first real run, which is a better argument for fixing the job than anything in the commit before this one. The M57 entry now names the version it needs, and the GitHub Release body is regenerated from the corrected section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The version-matrix job has not run the matrix since the commit that first produced the table, and reported
success every time.
What was happening
Gradle restored
:kdrant-transport-rest:jvmTestFROM-CACHE on every run after the one that first wrotethe report. The test never executed, wrote no table, uploaded no artifact, and the job was green throughout.
if-no-files-found: ignoreis what kept the missing upload quiet.The README's table is the evidence. It names Qdrant
v1.19.0and 39 contract cases; the pin isv1.19.1and the contract has 46. Three releases stale, and nothing anywhere went red.
That is from the run on
2874572, which merged an hour ago.What changes
--rerunon the task. This job's product is a measurement, not a verdict, and Gradle cannot tell thedifference: the inputs had not changed, so the cached result was a correct answer to the wrong question.
A missing table is an upload error. The two failures here are not the same and were being treated as
one. A cell that fails against an older server is exactly what this job exists to publish and must not fail
it — that is why the job is
continue-on-errorand why the test counts instead of throwing. A run thatproduced no table measured nothing, which is a defect.
workflow_dispatchon the workflow. So the matrix can be re-measured without pushing a commit. Qdrantreleasing a minor is the usual reason to want that, and until now the only way to ask was to invent a push.
The table itself
This PR's own CI run produces it, and the real numbers go into the README in this branch before it merges.
The table is not hand-edited: the only reason it is touched here is that the job which writes it has not
been able to.