Skip to content

feat(coverage): coverage table in the app's JVM; flag tests a job ran in - #194

Draft
slayerjain wants to merge 18 commits into
feat/dedup-sdk-220from
feat/jcov-buildtable
Draft

slayerjain wants to merge 18 commits into
feat/dedup-sdk-220from
feat/jcov-buildtable

Conversation

@slayerjain

@slayerjain slayerjain commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

What this branch does today

Stacked on #193 (which is stacked on #192).

Keploy's per-test code coverage for Java needs the app's build table (packages, classes, methods, their lines and branch edges) and, for each class, which of those leaves a hit of each JaCoCo probe covers. Only the app's JVM holds the class bytes in every run mode, so keploy-sdk builds both there, and no class file leaves the app.

When it runs

  • Only when Keploy asks, with the control-socket command TABLE, which Keploy sends once its first START is answered. The SDK answers ACK proto=2 at once and works on a daemon thread of its own: START and END never analyze, encode or send. An older Keploy never sends TABLE, so it never gets a frame; an older SDK answers it IGNORED.
  • A TABLE while a send runs is logged at INFO and has that send run once more after it, so a new receiver can ask again. A request made during a send that an interrupt ended runs on a new thread.
  • Before it starts, the SDK reads the free heap: the JVM's heap cap less the heap in use after its newest collection (it never runs a collection). With less free than 64 MiB plus 1 KiB per indexed class (113 MiB at 50,000 classes) it answers IGNORED proto=2 reason=low_heap heap_free_mb=F heap_needed_mb=N, logs a WARNING and sends nothing, so Keploy can mark the run not measured with that reason. KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB (else -Dkeploy.coverage.table.min.heap.mb) sets the bound in MiB; 0 turns the check off.
  • Only the unix control socket serves TABLE. The TCP transport sends no table.

The table

  • One JaCoCo pass per class file gives both the table and the probe map. The analyzer is the JaCoCo core keploy-sdk.jar carries (0.8.15, relocated).

  • One record per top-level class, with its nested, inner, anonymous, lambda and Scala module classes folded in by name. A method has one code leaf per line and one leaf per branch edge.

  • Records are in the ingest contract's order and written as Keploy's Go encoder writes them. Golden rows check the bytes against the Go encoder's output on the java-dedup sample's classes and on generated classes.

  • The build id is jv1- and the first 32 hex of the sha256 over "keploy-java-table " F "\n" "jacoco " V "\n" followed by the sorted class ids (JaCoCo's CRC64, unsigned order, de-duplicated, 16 lowercase hex), joined by newlines. F is the table format (1) and V the analyzer's JaCoCo release. The INDEX record takes its build id from the same function, so the two always match, and an SDK release that writes a different table for the same classes names a different build.

  • A unit is not_measured, with no methods and no probe record, when:

    • analyzer_failed: JaCoCo cannot analyze it, or its probes cannot be mapped exactly (this includes two class files of one name whose methods or probes differ);
    • record_too_large: its table record is over 8 MiB, or it is too complex to map: its probe map is over 8 MiB or 64 Mi walk steps (a method of about 3,000 early returns, as generated parsers have). The SDK's INFO line counts the two apart.

    Keploy leaves such a unit's lines out of the totals and never counts them as missed. The rest of the build is measured as usual.

The probe map

  • One record per class name of each measured unit: the class's JaCoCo id, its unit, and for each probe the leaves it covers, as ranges of indices into the unit's leaves. A probe covers every instruction on the path that ends at it and the branch edges along that path, so a test's leaves are the union of its hit probes' leaves. The map is never uploaded.
  • It is read off JaCoCo's own analysis: a small class in JaCoCo's package runs its package-private steps unchanged and checks every line against the coverage JaCoCo computes in the same pass. An oracle row compares each probe alone, every pair and all of them with JaCoCo's public Analyzer, on the java-dedup sample, JaCoCo core, Gson, and Java 8 and Java 21 code shapes.

On the wire

  • TABLE frames on the data socket, one frame per connection, each at most 1 MiB: KTB1, a 4-byte big-endian length, a JSON header line ({"v":1,"part":"table","build":"jv1-...","seq":k,"last":false}; the last frame adds records and the stream's sha256), then the next bytes of the NDJSON stream. The probe map follows the table's last frame as a second stream, "part":"probes", seq from 0.
  • After each frame the SDK waits for OK. A receiver that closes without it ends the send, with a WARNING naming the frame, and the table goes again only when Keploy asks again.

Memory and CPU

  • Records stream as they are made. The probe map waits in an owner-only temp file in java.io.tmpdir, unlinked once open on Unix, and is streamed after the table. A send holds the class list, one frame and one class's analysis. A tmpdir the app cannot write ends the send before any frame, with a WARNING naming it.
  • The send is paced by the JVM's collections, so its short-lived garbage does not make the collector grow the app's heap: it owes 200 ms for each ms of collection while it sends. Every wait is drawn from the send's own work time (at most a second saved up), so a paced table takes at most about twice as long as unpaced, whatever the app collects, and frames keep coming. No wait may push the table's end past 20 s at 50,000 class files (0.4 ms a class file past that).
  • Measured at 50,000 generated classes, on JDK 8, 17 and 21, G1 and Parallel, in 1 and 4 GiB pods:
    • a quiet app: the table took 5.0-8.1 s (2.5-4.0 s of it paced), 2.4-3.7 s of thread CPU, and at most 0.74-1.21 s between two frames;
    • a busy app (two threads allocating 400 MB/s): 4.9-8.5 s, each send's waits within its own 2.5-4.3 s of work.
  • A method of 3,000 if/else branches maps in about 0.2 s of CPU.

Logs

  • One INFO line per send: the classes measured, those not measured by reason, and the class files built without line numbers (compile with -g).
  • A WARNING when the table is declined or a send fails; an INFO line when a request waits for a running send, and when the send is first paced.
  • If the JVM exits while a send runs, one line on System.err says how far it got (class files analyzed, frames sent, time and how much of it was paced). A JVM that is killed (SIGKILL) prints nothing.

Scheduled jobs in a test's window

  • JaCoCo's probes do not say which thread hit them, so a Quartz job or a Spring @Scheduled method that runs while a test runs puts its lines in that test. The SDK now names such jobs in the test's record: "contaminated":["quartz:<group>.<name>" | "scheduled:<class>.<method>", ...] after windowCut, at most 8 names then "+N", left out when none ran (unix data socket and TCP COV line alike). The probes stay in the record; Keploy does not judge such a test.
  • A run counts in a test when it was going on at the test's START (the window opens before the START clears the probes) or started before its END read the hits. A run that ended before the START does not: the START clears what it ran.
  • How: org.quartz.core.JobRunShell.run() (Quartz 1.x and 2.x, and Spring's Quartz support) and Spring's ScheduledMethodRunnable.run() are changed as their classes load, the way a server's request method is: the job's own code moves to a private keploy$job$run, and the method of the old name marks the run around it, however it returns or throws. Same transformer, so the same guarantees: after JaCoCo's transformer whatever the -javaagent order, made again when another agent retransforms the class, nothing when the class is never loaded. Job runs count whatever KEPLOY_JAVA_END_WAIT_MS says.
  • Cost: one lock at each START and END, two per job run, and per job run a reflective read of its name (microseconds), whether or not a test runs; no per-test thread scan. The lock is held only to add or remove a run and to swap the window, so many scheduler threads firing at once wait on each other for that long.

A test's own Spring async work

  • A request that hands work to a Spring executor (Spring 5.3 to 6.2; an event listener on an async multicaster, as Apache Fineract runs its events on a SimpleAsyncTaskExecutor; a task given to a ThreadPoolTaskExecutor; an @Async method) returns before that work is done. Work that outlived the test's END landed in the next test's window, and that test's START cleared it as background hits. That work now counts as the request's: END waits for it under the bound it already waits for the request (KEPLOY_JAVA_END_WAIT_MS, 200 ms by default, at most 2,000 ms), and a test whose work still runs at the bound says windowCut, so Keploy does not judge it. The record has no new field.
  • Only work handed off on a thread that runs a counted request (Tomcat, Jetty, Undertow), or on a thread that runs such work, counts. Work a scheduler's thread or one the app starts hands off is not waited for.
  • How: SimpleAsyncTaskExecutor.doExecute (which each of its execute and submit methods ends in), ThreadPoolTaskExecutor's execute, submit and submitListenable (its submitCompletable, from Spring 6, is AsyncTaskExecutor's default, which goes through execute), and AsyncExecutionAspectSupport.doSubmit (each @Async call, whatever its executor) are changed as their classes load, by the transformer that changes the servers' request methods, with the same guarantees. The method's own code moves to a private keploy$async$<name>, and the method of the old name passes its task to the SDK, which counts it until it returns or throws. A task the executor throws on (rejected, shut down) is not waited for, and a task that is a Future stays one. The rows run on Spring 5.3 and, on Java 17 and later, on Spring 6.2 (Spring Boot 3.4's), including 6.1's SimpleAsyncTaskScheduler, whose poller is not waited for. Nothing changes when the wait is off (KEPLOY_JAVA_END_WAIT_MS=0).
  • Cost: per request, a thread-local set and cleared; per task handed off, one small object and three atomic operations. Per test (START to its ACK plus END to its ACK, median of 200 tests, in each of three runs), with the shaded jar on 50,000 generated classes over the TCP channel and each request counted: 1.62-1.81 ms with this change, 1.49-2.13 ms without it, 1.40-2.02 ms when each request also hands a task to a plain pool through the SDK's hand-off, all with TCP_NODELAY on the SDK's socket, which this branch now sets (below).

Replies over the TCP channel

  • Over the TCP channel (the k8s path) the SDK answered END in two writes, the COV line and then the ACK line. Keploy reads up to the ACK before it sends the next command, so its kernel delays its acknowledgement of the COV (about 40 ms on Linux), and until then Nagle's algorithm held the ACK line in the app's kernel: about 40 ms per test. The socket to Keploy is now opened with TCP_NODELAY (if the option cannot be set, the SDK logs that at INFO and goes on), and each reply is built in a 64 KiB buffer and sent in one write. A frame larger than that (the first END's CLASSES frame, a MANIFEST frame once the app has hit several hundred classes, a very large COV) goes to the socket as it is, without a copy, and TCP_NODELAY sends what follows it at once. The frames and their order are as before; the diagnostics line now lists END's steps as json, classes, send. END writes its COV into the reply before it builds the CLASSES frame, so a failure building that frame skips only that frame, never the COV; the classes step therefore also counts adding the COV to the reply (for a COV larger than the buffer, its write to the socket).
  • Per test, with the shaded jar on 50,000 generated classes, the SDK's own JaCoCo, the TCP channel on loopback and no KEPLOY_BUILD_TAG (so no CLASSES or MANIFEST frames), median of 200 tests in each of three runs: END to its ACK took 42.6-45.8 ms before and takes 0.73-1.21 ms now; START to its ACK plus END to its ACK took 43.9-47.7 ms and takes 1.22-2.14 ms.
  • The unix sockets have no Nagle, and their replies were already one write.

Limits (fail closed)

  • A table reaches Keploy only from a JVM that runs until it is sent. A native keploy test restarts the app at each test set; a set that ends before its table arrives is not measured (its tests are kept), and the next JVM sends the table again.
  • The committed heap still grows while the table is built. At 50,000 classes, over the heap after a full collection, on a quiet app: G1 11-79 MiB in 1 GiB pods and 59-300 MiB in 4 GiB pods (JDK 8: 300 MiB), Parallel 23-54 MiB. The committed heap never passes -Xmx, so this matters only for a pod whose -Xmx is larger than its memory limit.
  • The probe map is JaCoCo 0.8.15's probe layout, that of the agent keploy-sdk.jar starts. It does not fit the hits of a JaCoCo agent of another version that the app's own command attaches (the INDEX record names that case).
  • Two class files of one name in the index are both charged against the probe map's bound (8 MiB or 64 Mi steps), so such a unit is not measured as too complex to map once its two maps together pass it, though its one record would fit.
  • A class too complex to map is sent as record_too_large, as Keploy's readers (the CLI and the server) accept only analyzer_failed and record_too_large as a table reason. Keploy's reason text for it says the class is too large to record: in this release that text also covers a class too complex to map.
  • Class files built without line numbers have no leaves, and are counted in the INFO line.
  • Scheduled work is seen only through Quartz's JobRunShell.run() and Spring's ScheduledMethodRunnable.run(). Work on threads the app starts itself, a custom executor or a java.util.Timer, a message listener, a Spring task registered as a plain Runnable (SchedulingConfigurer), a reactive (Publisher-returning) or Kotlin suspend @Scheduled method (Spring 6.1+ runs those through another class), and a test's own async work that runs past its END on an executor other than those below are not seen; their lines stay in the test as before. Code a job's run reaches outside those two methods is not counted either: a Quartz job's creation (a JobFactory, or Spring's autowiring of the job) and a Spring ErrorHandler called after a @Scheduled method threw. A job that runs for the whole test run (a polling loop) names every test, so none of them is judged.
  • A test's own async work is waited for only on Spring's SimpleAsyncTaskExecutor, ThreadPoolTaskExecutor and @Async. Work handed to a plain java.util.concurrent executor, a thread or Timer the app starts, a ThreadPoolTaskScheduler's execute, or a message listener, or handed off by a WebFlux (Netty) request, is not waited for, and lands after END as before. A task the executor drops without throwing (a pool with a DiscardPolicy or DiscardOldestPolicy), and a task whose Future is cancelled or completed before it starts (in a ThreadPoolTaskExecutor's queue, or an @Async method's Future or CompletableFuture that the caller cancels or completes, as orTimeout does), hold its test's wait to the bound, and that test says windowCut. An executor subclass that overrides the changed methods without calling them is not counted.
  • No job run is seen when keploy-sdk.jar was not started as a -javaagent (attached to a running JVM, or started from the app's code); the SDK logs that once at INFO. A scheduler class that loaded before the agent started (loaded by another agent) is not changed either, and nothing logs it.
  • A JaCoCo reached over TCP (KEPLOY_JACOCO_PORT, the fallback when its in-process runtime is not found) is read as before: its dump can still wait about 40 ms per test on a delayed ACK (seen against a stand-in, not measured with JaCoCo's own tcpserver). A JaCoCo read in the JVM (the SDK's own, or the app's through its in-process runtime) has no such wait.

Part of keploy/enterprise#3435 (code coverage and dynamic dedup for large Java apps). Draft: this branch is updated at milestones while the work lands; it is marked ready for review when its checks pass. Do not merge before then.

@slayerjain
slayerjain force-pushed the feat/jcov-buildtable branch 2 times, most recently from b449cc6 to 7d8d216 Compare November 20, 2026 12:26
@slayerjain slayerjain changed the title feat(coverage): build the app's class and method map inside the JVM feat(coverage): build the coverage table and probe map in the app's JVM Nov 20, 2026
@slayerjain
slayerjain changed the base branch from feat/java-dedup-app-classes to feat/dedup-sdk-220 November 20, 2026 12:26
Keploy's per-test code coverage needs the build table of the app's
classes: packages, classes, methods, their lines and branch edges. Only
the app's JVM holds the class bytes in every run mode, so keploy-sdk
builds it there.

- The table goes only when Keploy asks for it with the control-socket
  command TABLE, after its first START is answered ACK. The agent
  answers ACK at once and starts the work on a daemon thread of its own;
  the control path never analyzes, encodes or sends, and START and END
  never ask for the table. An older Keploy never sends TABLE, so it
  never gets a frame; an older SDK answers it IGNORED. A TABLE while a
  send runs has that send run once more after it, so a new receiver can
  ask again. A TABLE that fails to start (no thread, or any error on the
  control path) is answered IGNORED and logged at WARNING, and the
  control server goes on serving START and END.
- The analysis competes with the app for CPU: the thread's priority is
  only a hint, which HotSpot on Linux ignores by default. When the index
  is not loaded yet, its first load runs on this thread and holds the
  index's lock, which an END that needs the index waits on; an END would
  otherwise load it itself.
- JaCoCo's analyzer runs over the indexed class bytes. One record per
  top-level class, its nested, inner, anonymous, lambda and Scala module
  classes folded in by name (the simple name cut at the first $ after
  its first character). A class is named by its bytes, the name JaCoCo's
  runtime gives its probes, wherever the index found the file. A method
  has one code leaf per line (n = JaCoCo's instructions on that line in
  that method) and one leaf per branch edge. A method with no line
  numbers has no leaf and is left out; the INFO line counts the class
  files built without line numbers and says to compile them with -g.
- Records are in the ingest contract's order (packages segment by
  segment, then id; methods by first line, then id) and written as Go's
  encoding/json writes them with HTML escaping off. The build id is jv1-
  and the first 32 hex of the sha256 over the sorted JaCoCo class ids of
  every indexed class; a record's hash is the same sum over its class
  files' ids.
- A class JaCoCo cannot analyze makes its record not_measured with
  reason analyzer_failed and no methods. A record larger than an
  inflated 8 MiB chunk is not_measured with reason record_too_large.
  Keploy's encoder also refuses a record whose compressed form passes a
  chunk; the CLI is the authority on that, so the stream's sha256 is the
  CLI's only while it re-marks no record.
- The records go to Keploy as TABLE frames on the data socket, one frame
  per connection, each at most 1 MiB: KTB1, a 4-byte length, a header
  line (build id, seq, last; the last one adds the record count and the
  sha256 of the whole stream), then the next bytes of the NDJSON stream.
  With no app class the stream is one empty last frame with build "".
  The SDK shuts its output after each frame and waits for OK; a receiver
  that closes without it ends the stream, which the SDK logs at WARNING
  with the frame and the reason, and the table goes again only when
  Keploy asks again.
- KeployDedupAgent gains only the TABLE branch in the unix control
  socket's command handler. The TCP transport does not send a table.

Two golden rows check the table byte for byte against the Go encoder's
output for the same classes, with the hashes and the build id computed
in Go from the class bytes: the java-dedup sample's classes, and a
generated set for the names, packages and escapes the sample lacks. A
control-socket row runs START and END on an index of the sample's
classes, checks that neither asked for the table, then sends TABLE and
gets that golden in one frame.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Keploy gets each test's coverage as JaCoCo probe hits per class, while
the build table counts lines and branch edges (its leaves). To turn hits
into leaves without the class files, Keploy needs, per class, which
leaves a hit of each probe covers. keploy-sdk now computes that map in
the app's JVM, with the same JaCoCo analysis steps as the table, and
sends it right after the table, in the same answer to TABLE.

- A probe covers more than its own line: JaCoCo marks every instruction
  on the path that ends at the probe, and the branch edges along it. So
  each probe maps to a list of leaves, and a test's leaves are the union
  of its hit probes' leaves.
- The map is JaCoCo's own result, not a copy of its rules. JaCoCo's
  public API gives only per-line sums, which lose which branch edge of a
  line is covered, and two tests that cover different edges of one line
  must count two. A small class in JaCoCo's package
  (org.jacoco.core.internal.analysis.KeployBranchCoverage) runs JaCoCo's
  package-private analysis steps unchanged (the instruction graph, the
  filters, merges and branch replacements) and reads each branch through
  JaCoCo's own replaceBranches. It checks every line's sums against the
  coverage JaCoCo computes in the same pass and throws on a difference.
  keploy-sdk.jar relocates org.jacoco.core, and this class moves with
  it.
- A method's probes never reach another method, so one analysis maps
  the k-th probe of every method: the map takes one analysis of a class
  file more than its largest method has probes, on top of the table's.
  Each one reads the whole class file, so a class with one very large
  method costs seconds of CPU on the analysis thread.
- The map is a second stream of TABLE frames after the table's last
  frame: "part":"probes", seq from 0, the last frame with the record
  count and the sha256 of the stream. One record per class name of each
  measured class: its name, its JaCoCo class id, its unit, and for each
  probe the covered leaves as ranges of indices into the unit's leaves.
  Indices are relative to the unit, so a unit Keploy marks not measured
  only drops its records. The map is never uploaded.
- A test's hits name a class by its name only, so a name has one
  record. The index can find one name under two paths: the same bytes
  count once, and two class files that differ share the record when
  their probes map alike (the record carries the first one's id).
- The map is JaCoCo 0.8.15's probe layout, that of the agent
  keploy-sdk.jar starts. It is not for the hits of a JaCoCo agent the
  app's command attached (the index record names that case): another
  version can place the same number of probes differently.
- A class whose probes cannot be mapped exactly is sent as not measured
  (analyzer_failed) in the table itself, so the table and the map always
  agree; that includes two class files of one name whose methods or
  probes differ. A class that is not measured has no map record, and the
  INFO summary counts the classes that failed this way.
- Lines below 1 have no leaf (javac never writes line 0; bytecode tools
  can): a method keeps its other lines in the table and the map.

Rows: for every class file of the java-dedup sample, of a Java 8 class
of code shapes (&& on one line, loops with break and continue, an int
switch with shared and fall-through cases, a String and an enum switch,
try/finally, try-with-resources, synchronized, a lambda, an assert) and
of a Java 21 one (records, a sealed type, pattern switches with guards
and record patterns, an exhaustive enum switch, a String switch with
yield, instanceof patterns), the leaves the map gives no probe, each
probe alone, every pair of probes and all of them equal what JaCoCo's
public Analyzer counts for that set, line by line (code leaf hit, branch
leaves hit, branch leaves), and each record has as many probes as
JaCoCo's instrumenter gives the class. The java-dedup map is a golden
for the Go decoder, and the control-socket row checks it arrives after
the table on a real socket. Rows cover one class file found twice, two
class files of one name that map alike and that differ (moved line,
missing method, probes in another order), and a method with line 0.

Only BuildTable.java changes in the agent; KeployDedupAgent.java is
untouched.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Keploy files the build table under the build id the INDEX record named,
so the two ids must be one value over one class set. Until now they were
two copies of the same algorithm: IndexReport.buildId for the INDEX
record and BuildTable's own classIdsSum for the table. A change to one
copy would have left Keploy looking for a table that never arrives.

- IndexReport.classIdsSum is now the only implementation: the sha256, as
  64 hex, over the class ids sorted as unsigned numbers, de-duplicated,
  each as 16 hex, joined by newlines. IndexReport.buildId is "jv1-" and
  its first 32 hex, or "" when there is no class.
- BuildTable.write takes its class ids with IndexReport.classId and its
  build id with IndexReport.buildId, and a unit's hash is
  IndexReport.classIdsSum over that unit's ids. BuildTable's copy and
  its sha256Hex helper are removed (tests now take the sha256 digest
  themselves).
- Both ids are taken over CoverageIndex.entries(), every class the index
  holds, whether a class reader can read it or not. Every entry has its
  bytes (the dir, jar and derived readers never store an entry without
  them), so the table's skip of an entry with no bytes never drops one
  the INDEX record counted.

The table bytes are unchanged: the Go encoder's golden for the
java-dedup sample still matches byte for byte.

Rows:
- ControlSocketTest.theIndexRecordAndTheTableNameOneBuild: the agent's
  index is the java-dedup sample plus one file no class reader reads;
  the INDEX record's build, and the build in every table and probe-map
  frame sent for TABLE, equal the id the test computes itself (sorted
  16-hex ids, sha256), which differs from the sample's alone.
- IndexReportTest.theUnitHashIsTheWholeSumAndTheBuildIdItsFirstHalf: the
  sum of the ids 1, 0xff and 0xffffffffffffffff, given out of order and
  with a repeat, is the full sha256 the Go side's build id is cut from.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The probe map analyzed each class once per probe round: a class with a
method of K probes was parsed, sanitized, flow-analyzed and filtered K
times, whole.

KeployBranchCoverage now analyzes a class once. For each method it keeps
the filters' output, then replays only that method's instructions
through JaCoCo's own probe adapter, analyzer, instruction builder and
coverage calculator, once per probe, with that probe alone executed.
The result is still JaCoCo's analysis, not a copy of its rules, and a
replay that meets other probes than the class's analysis did fails the
class (not_measured, analyzer_failed). The filters now run once per
class, in class order, exactly as for the table. A method with no line
(a class built without -g) has no leaf to map, so it is not replayed.

A class now costs, summed over its methods, each method's probes times
that method's size. One harness, run before and after on the same
machine (JDK 21, heap capped at 1 GiB), on a generated app of 50,000
classes (one 15-branch method each): the table and probe map took
51.4 s of wall time and 58.3 s of process CPU before, 29.1 s and 33.5 s
after.

BuildTableBudgetTest measures it in the test JVM on JDK 8, 17 and 21,
heap capped at 1 GiB, table and probe map:
- 50,000 generated classes: 28.6-31.2 s wall, 27.4-30.3 s on the
  table's thread, 35.7-38.1 s of process CPU (GC 0.6-1.0 s). At most
  100-114 MB of heap stayed in use after any collection. Output: 55 MB
  of table and 21 MB of probe map. With the heap capped at 256 MB
  (JDK 21): 33-36 s wall, 41-64 s of process CPU (GC 1.9 s).
- One class with a single 2000-branch method: 12.1-15.1 s; with 3500
  branches (JDK 21), 57.4 s. No frame goes out while one class's map
  is computed, so such a class pauses the stream that long.
- JaCoCo core's jar (210 classes): 0.5-0.6 s.
The analysis runs on its own daemon thread after Keploy asks for it, and
START and END are answered while it runs.

Also:
- A TABLE that arrives while a send runs now runs the newest request
  after it, not the first one again. Each request names the index and
  data socket it was made for, so after the agent's stop() and start()
  during a send, the rerun used the old index and the old socket. A
  send that throws no longer drops the request made while it ran.
- BuildTable's logging can no longer throw. java.util.logging does not
  catch what a Handler throws (an app's logging bridge), so a log call
  that threw could fail a send, and one in the send loop left it marked
  running with no thread: every later TABLE was ACKed and never sent.
- keploy-sdk's tests run with a 1 GiB heap (the argLine property;
  -DargLine overrides it). Left to the JVM, the heap was a quarter of
  the machine's RAM. In a 4 GiB container where JDK 8 sizes it from
  the host's RAM, the new rows let its collector grow the heap until
  the kernel killed the test JVM.
- IndexReport.hex is the one lowercase-hex encoder (the build id and a
  frame's sha256); BuildTable's copy is removed.
- BuildTable.write no longer skips an index entry without bytes: the
  agent never passes one, and the INDEX record has no such skip.
- BuildTableTest's receiver keeps each frame before it answers OK, as
  Keploy does; it answered first, so a row could count one frame short.

Rows:
- ProbeMapTest.eachProbesReplayIsTheClassAnalyzedWithThatProbeExecuted:
  on JaCoCo core's and Gson's jars, each probe's replay equals the class
  analyzed whole with that round's probes executed, for every method
  and probe.
- ProbeMapTest.aMethodTheCallerDeclinesIsNotReplayed: a method the
  caller declines gets no replay; the class pass still analyzes it.
- BuildTableTest.theRunAfterASendIsTheNewestRequests: two requests
  during a send run the newest once (it ran the first one again).
- BuildTableTest.aRequestMadeWhileASendFailsStillRuns: a request made
  during a send that throws still runs (it was dropped).
- BuildTableTest.aLogHandlerThatThrowsNeitherFailsASendNorWedgesSends:
  with a log handler that throws, send does not throw, and a request
  after a failed send, and after a sent table, still runs.
- ControlSocketTest.startAndEndAreAnsweredWhileTheTableIsAnalyzed: with
  a 5000-class index, START/END pairs are answered while the table is
  analyzed and sent; TABLE's own answer and the slowest pair each take
  under 5 s.
- BuildTableBudgetTest: measures and prints the cost on a generated
  app, a one-method class and a real library; every class is measured.
  -Dkeploy.budget.classes and -Dkeploy.budget.branches set the sizes.
- The java-dedup table and probe-map goldens are unchanged.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The app's JVM builds the coverage build table and probe map when Keploy
asks (TABLE). The table already went out record by record, but every
probe-map record stayed in the heap until the table's last frame, as the
map streams after the table: about 25 MB more live heap for 50,000
classes, and more for classes with more probes. An app with a tight -Xmx
(a k8s pod) could run out of heap while the table was built.

Each measured class's probe records now go, as they are made, to a temp
file in java.io.tmpdir (owner-only, opened without following a link,
unlinked once open on Unix, so a JVM that dies mid-send leaves none and
one that dies before the open leaves an empty file), which is streamed
back as the probe map's frames after the table's last frame. The wire
is unchanged. Each class is dropped once written and the
grouping's lookups end before the analysis, so a send holds the class
list, one frame and one class's analysis. A tmpdir the app cannot write
ends the send before any frame, with a WARNING naming the dir: Keploy
gets no table for that build and asks again on its next run.

When TABLE arrives the SDK reads the free heap: the heap cap the JVM
reports (a little under -Xmx with the Serial and Parallel collectors)
less the heap in use. That is what the JVM's newest collection left in
the heap pools (the collectors' last GcInfo), or the heap in use now
when that is smaller, when no collection ran yet or when the runtime
does not report one. The SDK never runs a collection. With less free
than 256 MiB it answers

  IGNORED proto=2 reason=low_heap heap_free_mb=F heap_needed_mb=N

logs a WARNING and sends nothing, so Keploy can mark the run not
measured with that reason. KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB, else
-Dkeploy.coverage.table.min.heap.mb, sets the bound in MiB; 0 turns the
check off. A Keploy that does not read the reason sees IGNORED, as from
an SDK that sends no table.

Measured on 50,000 generated classes, against the parent commit in the
same run: the live heap at each frame (after a full collection) is
18-22 MB above the heap in use before the send (the classes' bytes, the
list of them the send is handed and the harness), from 44-47 MB (JDK 8
and 21); what is left is the send's own entry for each class, about
0.4 KB per class. Time is unchanged: 25.3-26.2 s wall,
24.7-25.4 s on the analysis thread at -Xmx1g (JDK 8/17/21). With
-Xmx96m it takes 44 s, from 71 s, and with -Xmx80m it completes in 48 s
where the parent ran out of heap (JDK 21).

Rows: a measured class's probe records reach their sink just before its
own table record; the map waits in a temp file that is gone during and
after the send, and both parts still match the java-dedup goldens; a
tmpdir that cannot be written sends no frame and names it; the temp file
is never opened through a link; the low_heap answer at and around the
bound, with 0 and no cap; the env, property and default of the bound;
the heap in use is the newest collection's, heap pools only, or the heap
in use now when smaller; and through the control socket, a TABLE with
too little heap is answered low_heap, starts no send and sends no frame
(it failed first: ACK). The other TABLE rows turn the bound off, so the
test JVM's -Xmx does not decide them. The budget row also reports the
heap delta, the pools' peak and, with -Dkeploy.budget.live=true, the
live set at each frame.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Each probe's leaves are now read off the class's one JaCoCo pass. JaCoCo
marks an executed probe's instruction and walks back through each
instruction's predecessor (the one edge into it that carries no probe;
JaCoCo puts a probe on every edge into an instruction that more than one
edge reaches), stopping after an instruction it marked already; its
calculator then merges and replaces branches as the filters said.
KeployBranchCoverage takes that walk over the edges the pass's builder
wired and applies the pass's filter output to what it marked. A method
costs its size plus its probes' walks, with no object per probe. Before,
each probe ran JaCoCo's builder and calculator over its whole method
again.

The walks are short in an if/else chain, whose statements join at labels
two edges reach. In a chain of early returns ("if (x == k) return k;"
lines, or a run of guard throws) each probe walks back over every check
before it and covers each one, so the map itself is quadratic: about
4.4 MB at 1,000 lines and 43 MB at 3,000, and at 4,500 it ran a 256 MiB
heap out of memory. A unit's probe map is now bounded at 8 MiB
(its records with their newlines, the table record's bound) and 64 Mi
walk steps. Past either, the unit is sent not_measured record_too_large
with no probe record and is counted in the send's log line, and the map
is given up as soon as it is past, so 3,000 and 4,500 lines now take
0.07-0.2 s of CPU and 40-51 MiB of allocation. The map's text is ASCII
in one buffer and copied once into an exact-size record. Two class files
of one name in the index are both charged against the one bound, so such
a unit is declined once its two maps together pass 8 MiB, though its one
record would fit (counted, never unbounded).

A method whose flow has another shape (an instruction that two edges
without a probe reach, or a replaced branch that reads another replaced
instruction) throws, so its class is not measured (analyzer_failed), as
a mismatch with JaCoCo's coverage did before.

The pass reads an instruction's branches only when it is covered (an
uncovered instruction has none; the table's pass executes no probe), and
keeps each instruction's branch numbers in one sorted array, not a set
per instruction.

The low_heap bound's default is 64 MiB plus 1 KiB per class the index
holds (113 MiB at 50,000 classes). 256 MiB declined every 1 GiB pod run
with no -Xmx. The count is the index's only when it is read already, so
the answer reads no class file and waits on no lock; an index not read
yet is bounded by the 64 MiB base, and read on the send's thread.

Measured on 50,000 generated classes (JDK 21 and 8, G1 and Parallel)
before the map's bound was added, which only takes copies out of the
map's path, so these are upper bounds: the send allocates 6,142-6,915
MiB, 125-141 KiB a class, against 90,611-98,076 MiB (about 1.9 MiB a
class) before, and takes 4.3-5.8 s of CPU against 29-42 s. A method of
3,000 if/else branches maps in about 0.2 s of CPU.

The heap the JVM commits while the table is built still follows the
analysis's allocation rate, as it did before (up to 657 MiB more with G1
on JDK 8 in a 4 GiB pod): the analysis is not paced, and this commit
does not bound that growth.

Rows: the probe oracle (each probe's map against JaCoCo's analysis of
the class with that probe executed) now covers JaCoCo core, Gson, the
java-dedup, code-shape and Java 21 fixtures, a 300-branch generated
class and an 80-return chain; the budget rows bound the allocation per
class (256 KiB), a 3,000-branch method's CPU (1 s), and a chain of
1,000, 3,000 and 4,500 early returns (mapped, then record_too_large,
each under 1 s of CPU and 64 MiB of allocation); the map's bytes and
steps bounds have rows at both edges; the heap bound has rows at both
edges of its default and through the TABLE answer with an index count
given and with none (-1).

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The build id hashed the class ids alone, so an SDK release that writes a
different table or probe map for unchanged classes named the same build
as the older SDK did. Keploy keeps one table per build, so every later
save of that build was refused (409 build_table_mismatch) until the app
itself was rebuilt.

The id is now "jv1-" and the first 32 hex of the sha256 over the ASCII
bytes

  "keploy-java-table " F "\n" "jacoco " V "\n" ID_1 "\n" ... ID_n

with no newline after the last id, where F is the table's format
(BuildTable.TABLE_FORMAT, 1) and V the release of the JaCoCo core whose
analyzer builds the table: the one keploy-sdk.jar carries renamed, read
from that core's own JaCoCo.VERSION less its build qualifier ("0.8.15"),
not the measuring agent's, since an attached agent of another version
only records the probes. The ids are as before: CRC64, sorted unsigned,
de-duplicated, 16 lowercase hex. These are the bytes the contract gives
Keploy's codecoverage.JavaBuildID. F covers the bounds that decide a
class's records (the record, probe-map bytes and probe-map steps
bounds) and the reason a unit past one is not measured, so a release
that changes one raises it. The frames that carry the records are not
in F: their header names its own version.

The INDEX record and the table still take the id from one function, and
a unit's hash is unchanged: the sum over that unit's class ids alone.

V is read once, at the first build id, from a holder of its own, so
the class-id helpers a testcase's END runs before its ACK never load
JaCoCo's version. If the renamed core cannot read its version resource,
V is "" (logged once) and the INDEX record and the table still name one
build. The smoke test, which CI runs on JDK 8, 17 and 21 against the
built jar, now fails a jar whose analyzer names a JaCoCo release other
than its org.jacoco.core's Maven version, read both as the SDK loads
from the class path and from the jar alone in a loader of its own. The
README says that an SDK upgrade that changes the table format or the
analyzer's JaCoCo gives the same classes a new build id.

Rows: the build id's three golden vectors (ids 1, 0xff and
0xffffffffffffffff with F=1 and V="0.8.15", with F=2, and with
V="0.8.14"), which differ from each other and from the id over the ids
alone; the SDK's F is 1 and its V the core jar's pom.properties version;
a JaCoCo version less its qualifier; the class-id helpers never load the
analyzer's JaCoCo version; and a tripwire that pins F=1 together with
the bounds, the step count of a 200-return chain, the sha256 of the
record each cap writes (a unit past the probe-map steps, and records
past a 1600-byte chunk) and of the golden table and probe-map records,
so a change to any of them, its reason code included, fails until F is
decided. The independent build id checks of the INDEX record and of the
table now hash the same bytes, and the java-dedup and edge goldens are
regenerated from an oracle that also reproduces their old ids-only
values. Their table and probe-map records are unchanged.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
JaCoCo analyzed every class file twice while the build table was made:
once through its public Analyzer, for the table record's methods, and
once through KeployBranchCoverage, for the probe map. The second pass
already runs the same steps (ClassAnalyzer's instruction graph,
filters and MethodCoverageCalculator), so the first was pure cost:
about 30% of what the send allocates and of its CPU.

Now one pass gives both. KeployBranchCoverage hands each method's
JaCoCo coverage (Method.coverage, computed as ClassAnalyzer computes
the coverage it adds to its class) to the caller before the method's
probes are walked, and BuildTable makes the table methods from it. A
probe's ranges are kept relative to its method and get the method's
base when the unit's records are written, since the base is known only
once all of the unit's methods are ordered. The map budget counts the
text before the bases are added (never more than the records hold),
then each whole record in its place, so the bound holds as before.

The table and probe-map records are byte for byte what they were: the
java-dedup and edge goldens and the records the build-id tripwire pins
pass unchanged. What changes:

- At 50,000 generated classes (G1 and Parallel; 1 and 4 GiB pods) the
  send's thread allocates 84-96 KiB a class (4,145-4,697 MiB in all;
  JDK 8, 17 and 21), from 123-134 KiB (6,021-6,577 MiB; JDK 8 and 21),
  in 2.5-3.4 s of CPU, from 3.5-4.8 s. The committed heap does not
  drop (G1 in 4 GiB pods grows it by 222-518 MiB, from 254-420): the
  collectors size it to the allocation rate, not to the garbage's
  total.
- A check of the map that fails (KeployBranchCoverage.NotMapped, or
  one of BuildTable's) counts as not mapped, as before; anything else
  the pass throws counts as JaCoCo failing to read the class (before,
  an error in the map's own pass counted as not mapped). Both are
  analyzer_failed. An error while a table method is made from its
  coverage now fails only its unit, as not mapped; before, it ended
  the whole send.
- A unit's class files are now all analyzed first, in name order, and
  the first analysis that fails decides; only then are its probe-map
  records made and charged to the budget. Before, every file's JaCoCo
  read came first, then each file's map and record in turn. So a unit
  with two failures of different kinds can now be given the other
  one's reason: analyzer_failed instead of record_too_large, or the
  reverse. Either way it is not measured and has no probe record.
- A second class file of one name is charged its text against the
  budget while it is analyzed, not its record, so the smallest bound
  that maps a unit of two such copies is a few hundred bytes lower.
- While a unit's records are made, only the class file being written
  keeps its ranges; a record whose text is not the length counted for
  it fails its unit as not mapped instead of being sent.

Rows:
- aGeneratedAppsTableIsMeasured bounds the allocation at 112 KiB a
  class, below what two analyses took (123-141 KiB in every run so
  far); on JDK 21 it fails on the two-pass code (135,287 bytes a class
  at its default 2,000 classes) and passes on this one on JDK 8, 17
  and 21.
- theTableNeedsNoSecondAnalysisOfAClass: no class of BuildTable's
  names JaCoCo's public Analyzer (the two-pass BuildTable does).
- eachTableMethodIsWhatJacocosAnalyzerCounts: on JaCoCo core's and
  Gson's classes and every fixture, with no probe executed and with
  all of them, each method's coverage from the one pass equals what
  the Analyzer counts (the same methods in the same order, every
  counter, line by line). It fails when the pass skips the filters.
- aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit now also adds
  a class JaCoCo's instrumenter wrote, and counts none of the three
  refusals as a map that failed (it fails when they are).

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
At 50,000 classes the build table's send allocates 4.2-4.8 GB in
2.5-6.3 s, nearly all of it short-lived. G1 and Parallel answer that
rate by committing more heap, and the app keeps that heap after the
send: unpaced, G1 on JDK 21 grew a 4 GiB pod's heap by 175-343 MiB.

Two changes:

- The send's thread is paced by the collections (BuildTable.Pacer).
  After each unit it owes 200 times the collection time since the send
  began (the app's collections and its own), less the time since, and
  sleeps that off, so collecting takes about 0.5% of the send's time.
  A JVM that does not collect is never paced. The sleeps come from an
  allowance of 0.8 ms per analyzed class file, of which at most 1 s is
  saved up: no one sleep is over a second, frames keep coming (so
  Keploy's wait for the next frame never runs out), and the sleeps add
  up to at most 0.8 ms a class file (40 s at 50,000). No sleep is taken
  that would end the table past 50 s at 50,000 class files (1 ms a
  class file past that), reckoned as the time so far plus the rest of
  the class files at the rate the work went so far, so on a slow or
  busy CPU the pacing gives way to the table's 60 s bound. A clock or
  collector bean that fails ends the pacing, and the rest is sent
  unpaced. An interrupt ends the send, with a WARNING that names it,
  and the interrupt is kept; before, the next write to the probe map's
  temp file failed and the WARNING blamed java.io.tmpdir. The send's
  INFO line now says how long it was paced.
- FrameWriter fills one frame buffer, reused for every frame, and the
  socket writes it in place. Before, each 1 MiB frame was allocated
  twice (the body's copy and the frame), as G1 humongous objects.
  FrameSender gains send(b, off, len); its default copies the bytes,
  for a sender that keeps them.

At 50,000 generated classes (G1 and Parallel; 1 and 4 GiB pods with no
-Xmx; JDK 8, 17 and 21), three runs paced against two of the same code
with the pacer off, the committed heap grows over what it was just
after a full collection by:

- JDK 21: 19-46 MiB (unpaced: G1 in a 4 GiB pod 175-343 MiB).
- JDK 17 and 8, on Parallel and on G1 in a 1 GiB pod: 11-92 MiB.
- G1 in a 4 GiB pod: 13-166 MiB on JDK 17, and 420-421 MiB on JDK 8 in
  every run (unpaced 422-707). The collection shrank G1's heap (JDK 8:
  632 to 120 MiB) and its pause raised G1's pause-time ratio, so G1
  grows the heap back while the table is sent, paced or not. Never
  past -Xmx.

The table takes 10.3-46.1 s, from 2.5-6.3 s, in 2.4-6.7 s of thread
CPU; the largest gap between two frames is 2.25 s. Where the pacing
does not cut the growth it still costs that time: Parallel on JDK 8
takes 28-46 s, from 3.6-4.1 s, for about the same growth.

Rows:
- BuildTablePacerTest (16 rows, on a clock the rows drive): no sleep
  without a collection; a collection is slept off to 200 times its
  time, less the work since; the sleeps over any run of units stay
  within a second plus the allowance of their class files; a quiet
  stretch saves up at most a second; an overrun sleep is charged, and
  a sleep the clock did not see is charged what it asked; the sleeps
  give way to the deadline, told the class files or not; a table whose
  work alone fills its deadline is not paced; an interrupt ends the
  send and is kept; a failing collector bean ends the pacing; write
  paces after each unit by its class files and tells the pacer how
  many come.
- anInterruptEndsTheSendAndIsSaidSo: an interrupt before the send, or
  after the table's last frame (when it closes the probe map's real
  temp file), ends the send with a WARNING that names it, the
  interrupt kept and no temp file left.
- aGeneratedAppsTableIsMeasured asserts the growth over the heap just
  after its full collection that the paced send reaches: under 128
  MiB, but under 192 MiB on G1 with a heap cap of 512 MiB or more on
  JDK 9-20 and under 448 MiB on JDK 8; the table within 60 s (scaled
  past 50,000 classes); and no frame more than 10 s after the one
  before. It fails with the pacer off (G1, JDK 21, 4 GiB pod: 175 and
  343 MiB). Its sink now reads each frame in place, counting a match
  split over two frames: a String of a 1 MiB frame is itself a
  humongous object, and grew the heap the row measures.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
An interrupt ends a build table send, and the interrupt is kept for
its thread. Before, Sends ran the request made during that send on the
same thread with the interrupt still set. That request names the
agent's newest index, which may not be read yet. On JDK 8 a class
file's read fails on an interrupted thread (ClosedByInterruptException),
so the index would keep only the classes it got, for the JVM's life (a
wrong build id). The request then ended at its first unit with a second
WARNING.

- Sends hands the request made during an interrupted send to a new
  thread, uninterrupted, and ends the interrupted one. Sends stays
  running while that thread starts, so a request made meanwhile waits
  to run after it. If that thread cannot start, the request is
  dropped (and any request made while it tried) with the same WARNING
  as a first thread that cannot start, and the next request starts a
  thread.
- send() checks the interrupt before it reads any class: on an
  interrupted thread it reads none, sends no frame, and logs the
  WARNING that names the interrupt.
- Pacer takes the analysis's rate from begin(), after the index's read,
  not from the send's start. Before, a slow index read counted as the
  analysis's work, so the rate was overestimated and the first units
  went unpaced. The 50 s deadline still counts from the send's start,
  the index's read included.

Rows: a request made while a send is interrupted runs on a new,
uninterrupted thread; one made while that thread starts starts no
thread of its own and runs after the first on that thread; one whose
new thread cannot start is dropped with a WARNING, as is one made while
it tried, and the next request runs; a send on an interrupted thread
reads no class; after a 10 s read the first wait comes as soon as the
allowance allows and the waits end by the deadline. Each fails on the
code before this change. The Pacer's interrupt row clears the flag in
a finally, so a failing row cannot leave it set for the rows after it.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A native keploy test stops the app at each test set's end. A build
table send still running then ended with no line in the app's log, so
a run whose table never arrived could not tell why from the app's side.
Two ways the SDK puts the table off were silent too.

- An exit notice: while a send runs, a shutdown hook is held for it and
  removed when the send ends. If the JVM exits first, the hook prints
  one line to System.err: how many class files were analyzed of how
  many (or that the class files were still being read), the frames
  sent, the time since the send began and how much of it was paced. It
  writes to System.err, not java.util.logging, because shutdown hooks
  run at once and in no order, and the logging handlers may already be
  closed. A JVM that is killed (SIGKILL) or halted runs no hook and
  prints nothing.
- A TABLE ask made while a send runs is logged at INFO, once per ask: it
  is sent again when that send ends, and an earlier ask that waited for
  it is dropped (the newest names Keploy's current socket).
- A paced send logs one INFO line at its first wait: the collection time
  since the send began, the wait per millisecond of it, the most the
  waits may add up to for its class files, and the deadline no wait may
  push the table's end past.

The Pacer now counts every analyzed class file, also when the pacing is
off, so the notice can say how far the send got.

Rows: a JVM that exits mid-send prints the notice exactly once; a JVM
whose send ends before its exit prints none (each in its own JVM); an
ask made during a send logs one INFO line each, the second naming the
dropped ask, and only the newest ask runs; a paced send logs one line
however many times it waits; a send that never waits logs none. The
first, third and fourth fail on the code before this change.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The build table's send was paced by owing 200 times all the collection
time since it began, the app's own collections included, and its waits
were bounded by 0.8 ms a class file (40 s at 50,000) and a 50 s
deadline. A busy app collects past that 0.5% goal for its own sake
(2.7% of the time while it allocates 400 MB/s), so every send in a busy
app waited out its whole allowance. A native keploy test restarts the
app at each test set, and each new JVM starts the send over: a
50,000-class app under test never got its table to Keploy within a set,
so the whole run had no coverage.

- The waits are drawn from the send's own work time, not from its class
  files: each unit's wall time since the one before (its socket writes
  included, its waits left out; the first unit's includes the index's
  read) is what the thread may wait, at most a second of it saved up.
  All the waits add up to at most the send's own work (plus one sleep's
  overrun), so a paced table takes at most about twice as long as
  unpaced, however much the app collects. No one wait is over a second,
  as before, so frames keep coming.
- No wait may push the table's end past 20 s at 50,000 class files
  (0.4 ms a class file past that), down from 50 s.
- The paced line says what is owed when the first wait is taken ("so it
  owes N ms now (each ms of collection adds 200, and each ms the send
  works or waits takes one off, never below 0)") and the bound.
- The exit notice says what it knows when the exit starts ("the JVM is
  exiting while its build table is being sent. So far: ..."): the send's
  daemon thread may still finish while the other hooks run. Its hook
  thread is made inside the try, so a notice that fails never ends the
  send, and a JVM already exiting when a send starts prints the line at
  once.
- When the thread for a request that waited out an interrupted send
  fails to start, the WARNING says that request is not sent.

Rows:
- A busy app's table: two threads allocate 400 MB/s, keeping a ring of
  arrays, while the table is sent. Its waits stay within the send's own
  work time (plus 0.5 s for one sleep's overrun), the table is whole,
  within 60 s, with no frame more than 10 s after the one before. 5,000
  classes by default, 50,000 with -Dkeploy.budget.classes. It fails on
  the code before this change: at 50,000 classes on JDK 21 with -Xmx4g
  and G1 the send took 43,498 ms, 39,991 of them paced, for 3,507 ms of
  work; at 5,000 classes, 3,990 ms paced for 655 ms of work.
- The Pacer rows are recast for the work bound: the waits add up to at
  most the work, and over any run of units to at most a second plus
  those units' time; a unit may wait its own time, not its class files;
  the deadline is 20 s; the Pacer counts its class files paced or not.
- A send removes its exit hook, after a whole send and after one whose
  socket fails (it fails if the hook is left registered). The hook the
  row reads is kept per thread, so the app keeps no send's buffers past
  the thread that sent it. The paced line names the owed time, after an
  index read longer than what the first unit owes, so the floor is hit
  and nothing carries over. The rows read only their own thread's log
  lines.

Measured at 50,000 generated classes, one run each, on JDK 21, 17 and 8,
G1 and Parallel, in 1 and 4 GiB pods (-XX:MaxRAM, no -Xmx):
- A quiet app: the table took 5.0-8.1 s (2.5-4.0 s of it paced; thread
  CPU 2.4-3.7 s; the longest gap between frames 0.74-1.21 s). Committed
  heap growth over the heap after a full collection, the row's bound in
  brackets: JDK 21 G1 20 and 59 MiB (128), Parallel 54 and 23 (128);
  JDK 17 G1 11 (128) and 145 (192), Parallel 33 and 32 (128); JDK 8 G1
  79 (128) and 300 (448), Parallel 47 and 32 (128), in 1 and 4 GiB pods.
- A busy app (the same cells, plus -Xmx4g G1 on each JDK): the table
  took 4.9-8.5 s, 2.4-4.2 s of it paced, each send's waits within its
  own 2.5-4.3 s of work. The committed heap grew 0-280 MiB in the pods
  and 0-816 MiB at -Xmx4g over the send's start; that is the app's own
  allocation as much as the send's, and it is not asserted.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A class whose probe map passes its bound (8 MiB of records, or 64 Mi
walk steps) has no probe map, so no test's leaves can be named in it.
The table sent it as not_measured record_too_large, the code for a
table record past 8 MiB, so Keploy could not tell a record too large
from a class too complex to map. It is now not_measured with the
contract's code unit_too_complex. It has no methods, as any
not_measured record, so Keploy leaves its lines out of the totals and
never counts them as missed; the rest of the build is measured as
usual.

- record_too_large is now only a table record past 8 MiB with its
  newline. The send's INFO line counts the two apart: "N not measured
  as too complex to map (unit_too_complex: a probe map over 8 MiB or
  64 Mi steps; ...)".
- The table format stays 1: no released SDK writes a table, so the
  build-id row's pinned hash of the capped record is updated alone, as
  that row says to do before the first release.
- A RuntimeException in Keploy's own steps of the class's one analysis
  (setting a method's instructions from JaCoCo's graph, and walking
  each probe; both under one wrap) is now NotMapped, with its cause.
  Before, an index error there was logged as "JaCoCo could not
  analyze" the class and was not counted with the probe maps that
  failed, though the class is the same analyzer_failed either way.
  JaCoCo's own steps (its instruction graph, filters and coverage
  calculator, with the hooks Keploy runs inside them) still throw as
  they do, and what the caller's callbacks (wants, and each probe)
  throw passes as it is, so a map past its bound stays
  unit_too_complex.

A row that a throwing log handler fails no send counted every record
published on the build table's logger, and expected two. A collection
during its first send adds the paced line (INFO), so it failed once
with three. It now counts the two sends' outcome lines, from its own
thread.

Rows: a probe map past its bytes or steps is unit_too_complex with no
probe record and not record_too_large; the send's line counts it
apart; a record past its bound is record_too_large and not
unit_too_complex; an index error in the probe walk is NotMapped naming
the class and method, while what the callbacks throw passes as it is.
Each failed on the previous code except two: the record-too-large row
is a guard (the previous code had no unit_too_complex to count), and
the throwing-handler row is the fix of a row that failed by timing.
In the NotMapped row only the walk's part failed there; its callback
parts guard against wrapping them, and since the walk and the setting
of instructions share one wrap, removing that wrap fails the row. The
early-returns budget row now counts unit_too_complex.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A class whose probe map passes its bound (8 MiB of records, or 64 Mi
walk steps) is sent as not_measured record_too_large again, as it was
before the previous commit, which sent it as unit_too_complex. Keploy's
readers (the CLI and the server's table ingest) accept only
analyzer_failed and record_too_large as a table reason, and refuse the
whole table over a record with any other code. So one class too
complex to map cost the run its coverage and dedup; with
record_too_large it costs that class only.

- The send's INFO line still counts such a class apart: "N not
  measured as too complex to map (record_too_large: a probe map over
  8 MiB or 64 Mi steps; the FINE log names each)". A table record past
  8 MiB is counted as before ("not measured as too large").
- Keploy's reason text for record_too_large says the class is too
  large; in this release it also covers a class too complex to map.
- Such a class's record is byte for byte what table format 1 wrote
  before the previous commit: the build-id row's pinned hash of the
  capped record is back to its earlier value.
- The rest of the previous commit stays: a RuntimeException in
  Keploy's own steps of a class's analysis is NotMapped, and the
  throwing-handler row counts only its own sends' outcome lines.

Rows: the build-id row (the capped record is record_too_large, its
pinned hash back), the bounds row (record_too_large at each bound,
counted as too complex to map and not as too large), the send's line
row (the class counted as too complex to map under record_too_large,
with no unit_too_complex and no "not measured as too large"), and the
early-returns budget row (counts record_too_large on the wire). Each
failed on the previous commit.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
JaCoCo's probes do not say which thread hit them, so the lines a Quartz
job or a Spring @scheduled method runs while a test runs land in that
test's coverage. Two replays of one test set then give different
duplicates, and a job's lines in one test can make another test that ran
that code look like its subset and be dropped.

The agent now sees each job run and names the jobs that overlapped a
test in its record, so Keploy does not judge that test:

- ServerEntries changes org.quartz.core.JobRunShell.run() (Quartz 1.x
  and 2.x, and Spring's Quartz support) and Spring's
  ScheduledMethodRunnable.run() (each @scheduled method) as their classes
  load, the way it changes a server's request method: the job's own code
  moves to a private keploy$job$run, and the method of the old name calls
  JobGate.enter(this, kind) before it and JobGate.exit after it, however
  it returns or throws. It runs after transformers that cannot
  retransform (JaCoCo's) whatever the -javaagent order, changes the class
  again when another agent retransforms it, and does nothing when the
  class is never loaded. Job runs are counted whatever
  KEPLOY_JAVA_END_WAIT_MS says. An agent not started as a -javaagent
  changes no class, and logs once that it sees no job run.
- JobGate keeps the runs going on. A test's START opens its window
  before it clears the probes, and each run going on then counts in it;
  a run that starts before the END has read the hits counts too; a run
  that ended before the START does not (the START clears what it ran).
  Both take one lock: a START and an END once each, a job run twice.
  A run also reads its job's name by reflection (microseconds). An
  Error after a run is counted as going on still hands its token
  back, so its exit counts it done and no later test is marked.
- The record (unix data socket and TCP COV line) gets
  "contaminated":["quartz:<group>.<name>" | "scheduled:<class>.<method>"]
  after windowCut: at most 8 names, then "+N" for N more; left out when
  none ran. The probes stay in it. A job whose name cannot be read is
  named by its kind and class, and still counts.

Rows: JobTailTest runs an app with a Quartz scheduler and a Spring
context with an @scheduled method, with the agent as its -javaagent:
runs inside a test (fired in the window, and one already running at the
START) are named, and runs between two tests are in neither. On the
commit before this one the first of those records has the job's hits and
no contaminated key. An app with neither scheduler on its class path
runs as before, and with KEPLOY_JAVA_END_WAIT_MS=0 both jobs are still
named. JobGateTest covers the window rules, the names, the 8-name cap
and a run whose count fails midway; ServerEntriesTest the changed
methods (returning and throwing, retransformed, END's wait off, a loader
that does not see the agent); ControlSocketTest and DedupWireFormatTest
the record on the unix socket, its key order and the log of an agent
started with no instrumentation. Quartz 2.3.2 and Spring 5.3.39 are test
dependencies only.

Limits: work on threads the app starts itself, a custom executor or a
java.util.Timer, a message listener, a Spring task registered as a plain
Runnable, a reactive or Kotlin suspend @scheduled method (Spring 6.1+
runs those through another class), and a test's own async work that
runs past its END are not seen. Nor is a Quartz job's creation (a
JobFactory, Spring's autowiring) or a Spring ErrorHandler called after a
@scheduled method threw, nor any run when the agent was not started as
a -javaagent or a scheduler class loaded before it. A job that runs for
the whole test run (a polling loop) names every test.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A request that hands work to a Spring executor (an event listener on an
async multicaster, as Apache Fineract runs its events on a
SimpleAsyncTaskExecutor; a task given to a ThreadPoolTaskExecutor; an
@async method) returns before that work is done. When the work outlives
the test's END, its lines land in the next test's window, where the
next START clears them as background hits: the test that ran the code
does not have it, and two replays of one test set can give different
duplicates.

That work now counts as the request's, and END waits for it under the
bound it already waits for the request (KEPLOY_JAVA_END_WAIT_MS, 200 ms
by default, at most 2000). A test whose work still runs at the bound
says windowCut, so Keploy does not judge it. The record has no new
field.

- RequestGate keeps the window a thread counts in, in a thread-local:
  enter() sets it on a request's thread and exit() clears it (a request
  inside a request gets its outer window back). RequestGate.hand(task),
  on a thread with a window, counts the task in that window from the
  hand-off until the task returns or throws, and the task runs with the
  window set, so what it hands off counts too. On any other thread (a
  scheduler's, one the app starts) the task is passed on as it is, so a
  scheduled job's hand-off is not waited for. enter() and hand() count
  last, so a failure before (out of memory) leaves nothing to undo. The
  same task object handed again (one executor method passing it to
  another) counts once; an executor that wraps it first counts the
  wrapper too, which ends when the task does. A task that is a Future
  stays one, for the executor's cancel at shutdown and its purge.
- ServerEntries changes, as their classes load and the way it changes a
  server's request method (the same transformer: after JaCoCo's whatever
  the -javaagent order, again when another agent retransforms the class,
  nothing when the class is never loaded, no Spring dependency):
  SimpleAsyncTaskExecutor.doExecute, which each of its execute and
  submit methods ends in; ThreadPoolTaskExecutor's execute, submit and
  submitListenable (its submitCompletable, from Spring 6, is
  AsyncTaskExecutor's default, which goes through execute); and
  AsyncExecutionAspectSupport.doSubmit, which hands each @async call to
  its executor, whatever that is. The method's own code moves to a
  private keploy$async$<name>, and the method of the old name passes its
  task through RequestGate.hand. When the executor throws instead of
  taking the task (rejected, shut down), the task is counted done.
  Nothing is changed when END's wait is off (KEPLOY_JAVA_END_WAIT_MS=0).

Cost: per request, a thread-local set and cleared; per task handed off,
one small object and three atomic operations (the count, the release's
compare-and-set and its decrement). Per test (START to its ACK plus END
to its ACK, median of 200 tests, in each of three runs), with the shaded
jar on 50,000 generated classes over the TCP channel and each test's
request counted as a server's: 1.62-1.81 ms with this change, 1.49-2.13
ms without it, and 1.40-2.02 ms when each request also hands a task to a
plain pool through RequestGate.hand, all with TCP_NODELAY set on the
SDK's socket. Without TCP_NODELAY every variant takes 44-48 ms, because
the SDK's reply waits for the reader's delayed ACK; this change does not
touch that.

Rows: ServerTailTest runs Tomcat with the agent as its -javaagent; a
request hands work that runs 500 ms after it to an async event
multicaster, a ThreadPoolTaskExecutor (submit, execute,
submitListenable), a SimpleAsyncTaskExecutor with a concurrency limit,
and an @async proxy over a plain java.util.concurrent pool, whose method
returns nothing or a CompletableFuture: each test's record has that
work's lines and no windowCut. Work that runs 4 s after its request,
past a 100 ms wait, makes the test windowCut. An event a scheduler's
poller publishes while a test's window is open is not waited for. The
first row runs on Spring 5.3; a second, on Java 17 and later, runs the
same on Spring 6.2 (Spring Boot 3.4's, whose jars a Java 17 build's test
profile fetches), plus the poller of Spring 6.1's
SimpleAsyncTaskScheduler, which starts each run in the changed doExecute
on its own trigger thread and is not waited for. On the commit before
this one the three Spring rows fail (the work's lines are not in the
record, and no test is cut); the row without Spring passes. An app
without Spring on its class path runs as before, and with the wait off
no executor is changed. AsyncHandTest covers RequestGate's counting
(nested requests, a task run on the request's own thread, a task that
throws, a rejected task, a Future, END's wait and its bound) and the
changed methods on stand-ins of the three classes at class versions 49
and 52 (returning, throwing, a method with no task left as it is,
retransformed). RequestGateTest's rows now exit each request on the
thread that entered it, as a server does, and check that the test thread
is left outside every window: three of them exited on another thread,
which left the test thread inside one and failed AsyncHandTest when it
ran after them. The README says what is waited for.

Limits: work handed to a plain java.util.concurrent executor, a thread
or Timer the app starts, a ThreadPoolTaskScheduler's execute, a message
listener, or from a WebFlux (Netty) request is not waited for, and lands
after END as before. A task the executor drops without throwing (a pool
with a DiscardPolicy or DiscardOldestPolicy), and a task whose Future is
cancelled or completed before it starts (in a ThreadPoolTaskExecutor's
queue, or an @async method's Future or CompletableFuture that the caller
cancels or completes, as orTimeout does), hold its test's wait to the
bound, and that test says windowCut. An executor subclass that overrides
the changed methods without calling them is not counted.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Over the TCP channel (the k8s path) the SDK answered END in two writes:
the COV line, then the ACK line. Keploy reads up to the ACK before it
sends the next command, so its kernel holds back its acknowledgement of
the COV until the delayed-ACK timer fires, about 40 ms on Linux. Until
then Nagle's algorithm kept the ACK line in the app's kernel, so every
test waited about 40 ms.

- The socket to Keploy is opened with TCP_NODELAY. If the option cannot
  be set, the SDK logs that at INFO and goes on.
- Each reply (START's ACK; END's COV, CLASSES and ACK; an IGNORED line
  and its ACK) is built in a 64 KiB buffer and sent in one write when
  its ACK is added. A frame larger than the buffer (the first END's
  CLASSES frame, a MANIFEST frame once the app has hit several hundred
  classes, a very large COV) goes to the socket as it is, without a
  copy, and TCP_NODELAY sends what follows it at once. Adding a line's
  newline no longer copies the line.
- END builds its COV and CLASSES frames before it writes either, so the
  diagnostics line now lists END's steps as json, classes, then send
  (the reply's writes). The frames and their order on the wire are as
  before.

The unix sockets have no Nagle, and their replies were already one
write. A JaCoCo reached over TCP (KEPLOY_JACOCO_PORT, the fallback when
its in-process runtime is not found) is not changed: its dump can still
wait about 40 ms on a delayed ACK (seen against a stand-in, not measured
with JaCoCo's own tcpserver).

Per test, with the shaded jar on 50,000 generated classes, the SDK's
own JaCoCo, the TCP channel on loopback and no KEPLOY_BUILD_TAG (so no
CLASSES or MANIFEST frames), median of 200 tests in each of three runs:
END to its ACK took 42.6-45.8 ms before and takes 0.73-1.21 ms now;
START to its ACK plus END to its ACK took 43.9-47.7 ms and takes
1.22-2.14 ms. TCP_NODELAY alone gives 0.81-1.22 ms and 1.32-2.09 ms on
the same runs.

Rows (CoverageTcpClientTest, Keploy's side played by the test on
loopback, JaCoCo's agent in the test's JVM; the three fail on the code
before this change):
- 25 START/END round trips: the median END round trip is under 20 ms.
- 25 rounds of START, END, and an END with no START: each reply comes
  whole in the one read that wakes on its first byte.
- The SDK's socket to Keploy has TCP_NODELAY set.
PhasesTest's TCP row now expects END's steps in their new order.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Over the TCP channel, END built its CLASSES frame before it wrote the
COV into its reply. Building that frame catches its own failures, but
anything there that threw would have skipped the COV, and the ACK, sent
in a finally, would have reached Keploy alone: the test then had no
record and was not marked unmeasured. END now writes the COV into the
reply first, so a failure building the CLASSES frame skips only that
frame.

A COV that fits the 64 KiB reply buffer only fills it, so a reply that
fits the buffer still leaves in one write, at its ACK. The frames and
their order on the wire are as before. The diagnostics line still lists
END's steps as json, classes, send; classes now also counts adding the
COV to the reply (a COV larger than the buffer is written to the socket
there).

The INFO line for a socket on which TCP_NODELAY cannot be set now names
the exception's class, as the connect-failure line does.

Rows (CoverageTcpClientTest, Keploy's side played by the test on
loopback, JaCoCo's agent in the test's JVM):
- An END whose CLASSES frame fails (reading the build tag throws) still
  sends its COV, then the ACK, and nothing else; the row also checks
  that the build tag's read threw. On the code before this change the
  reply is the ACK alone.
- A frame larger than the reply buffer (a CLASSES frame of about 137 KB,
  from 100 KB of class bytes that do not compress) arrives whole: its
  zip holds the class's bytes, the ACK follows on a line of its own, and
  the median of the 25 END round trips after it is under 20 ms. The
  SDK's thread that serves Keploy allocates under 16 MiB in that
  connection (about 2.7 MB on JDK 17 and 21); a reply buffer sized to
  hold a large app's CLASSES frame would go over it (a 64 MiB buffer:
  about 69 MB).
- The TCP_NODELAY row's assertion says what it checks.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain slayerjain changed the title feat(coverage): build the coverage table and probe map in the app's JVM feat(coverage): coverage table in the app's JVM; flag tests a job ran in Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant