Skip to content

Measure the claim, not the workflow around it - #164

Merged
tonytonycoder11 merged 2 commits into
mainfrom
fix/measure-the-claim-not-a-workflow
Sep 14, 2026
Merged

tonytonycoder11 merged 2 commits into
mainfrom
fix/measure-the-claim-not-a-workflow

Conversation

@tonytonycoder11

Copy link
Copy Markdown
Contributor

Fixes main, which I turned red in #163.

What actually went wrong

A run measured 290 ms against a usual 30 and failed the 75 ms ceiling I added yesterday. My first read
was that the ceiling was too tight. It was not: the measurement was measuring the wrong thing, and I had
put an assertion on it without reading what it covered.

The clock started here and stopped after the search:

val started = TimeSource.Monotonic.markNow()
...
if (qdrant.collectionExists(collection)) qdrant.deleteCollection(collection)
qdrant.createCollection(collection) { ... }
qdrant.upsert(collection, wait = true) { ... }   // blocks until the write is durable
val hits = qdrant.search(collection) { ... }
val elapsed = started.elapsedNow()

Five round trips, one of them a durable write. That number moves with the server's disk and its
optimiser, and the README was quoting it as how long a cold process takes to answer its first search. On
a loaded shared runner it can be 290 ms with nothing here being slower.

What it measures now

Process start, client construction, connecting, one search. The collection is seeded by the job with
curl before the binary runs, because a collection the process created itself would be inside its own
measurement.

The write path still runs, past the clock, on a separate collection. Those calls reach reflection the
read path does not, and an image proven only to read is not proven, so losing that coverage to tighten a
number would have been a bad trade.

The ceiling

150 rather than 75, and the comment says it is provisional. I set 75 from four samples that turned out to
be four samples of a narrower thing than I was measuring. It should be tightened once there are enough
runs of the new span to know its shape, rather than from another handful.

The README

The comparison row no longer carries a frozen figure. It says a cold process answers in tens of
milliseconds, measured and bounded on every push, which stays true after a fast runner or a slow one
without anyone editing it. The paragraph under the table says what the timed span covers, since that is
the part that made the old number misleading rather than merely stale.

A run came back at 290 ms against a usual 30 and failed the ceiling I added
yesterday. The ceiling was not the problem. The measurement was.

The timed span started before `collectionExists`, a `deleteCollection`, a
`createCollection` and an `upsert` with `wait = true`, and ended after the search.
Five round trips, one of which blocks until a write is durable. So the number moved
with the server's disk and the optimiser, and the README quoted it as how long a cold
process takes to answer its first search. On a loaded runner that span can be 290 ms
without anything in this repository being slower.

The clock now covers what the sentence claims: process start, client construction,
connecting, and one search against a collection the CI job seeded with curl before
the binary ran. The write path still runs, past the clock, on its own collection,
because those calls reach reflection the read path does not and an image proven only
to read is not proven.

The ceiling is 150 rather than 75, and the comment says why it is a guess: four
samples told me the distribution was 29 to 42 and they were four samples of a
narrower thing than I was measuring. It should be tightened when there are enough
runs of the new span to know its shape.

The README's comparison row drops the frozen figure. It says a cold process answers
in tens of milliseconds, measured and bounded on every push, which is true without
anyone editing it after a fast runner or a slow one.
@tonytonycoder11
tonytonycoder11 requested a review from a team as a code owner September 14, 2026 10:16
@trueup-by-nacode-studios

trueup-by-nacode-studios Bot commented Sep 14, 2026

Copy link
Copy Markdown

This pull request adds no capability. The code it touches can already read the environment (2 places).

What I could not follow

1 call resolve to a name whose type is written nowhere, so what they reach is not established here. Across everything it read that is 1 of 51, 2%.

  • System.getenv("QDRANT_PORT")?.toIntOrNull()example-native-image/src/main/kotlin/dev/kdrant/example/nativeimage/Smoke.kt:35

If one of those reaches the network or starts a process, this report does not show it.

3 files read · 2324 ms · against b5112d0 · engine 1.4.1

The first run of it reported 8 ms. 'Tens of milliseconds' was written before there
was a measurement of the new span and describes the old one, which had a server
creating a collection inside it. The table also now says why the figure fell, since a
number dropping by 4x with no code change is the kind of thing a reader is right to
distrust.
@tonytonycoder11
tonytonycoder11 merged commit 95e6656 into main Sep 14, 2026
14 checks passed
@tonytonycoder11
tonytonycoder11 deleted the fix/measure-the-claim-not-a-workflow branch September 14, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant