Skip to content

WebGL backend silently returns all zeros for long-running kernels (no GL error, no context loss) #859

Description

@fuzzie360

Summary

On the WebGL2 backend, a kernel that does enough per-thread work returns all zeros instead of its result. There is no exception, gl.getError() is 0, gl.isContextLost() is false, no webglcontextlost event fires, and the framebuffer reports FRAMEBUFFER_COMPLETE. The same kernel on the cpu backend returns the correct value.

The failure mode is what makes this worth reporting: the caller cannot tell a completed kernel from an abandoned one. Downstream code sees plausible-looking numbers (0) and blames its own arithmetic.

I suspect the underlying cause is the platform GPU watchdog abandoning a long-running draw rather than anything gpu.js computes incorrectly — but gpu.js currently surfaces that as a successful call, and that is the part that seems actionable.

Reproducer

Self-contained, no build step — save and open it. It loads gpu.js 2.20.0 from jsDelivr.

<!doctype html>
<meta charset="utf-8">
<title>gpu.js — silent all-zero result for long-running kernels</title>
<script src="https://cdn.jsdelivr.net/npm/gpu.js@2.20.0/dist/gpu-browser.min.js"></script>
<body>
<pre id="out">running…</pre>
<script>
// Per-thread work is controlled by an argument and the exact answer is known:
// acc starts at 1 and is incremented `trips` times by 1e-7. The `acc > 1e9`
// guard exists only to stop the loop being optimised away.
function makeKernel(gpu, n) {
  return gpu.createKernel(function (seed, trips) {
    let acc = seed[this.thread.x];
    for (let i = 0; i < 20000000; i++) {
      if (i >= trips) break;
      acc = acc + 1e-7 * (acc > 1e9 ? 0.0 : 1.0);
    }
    return acc;
  }, { output: [n], loopMaxIterations: 20000000 });
}

function trial(mode, n, trips) {
  const gpu = new GPU({ mode });
  const k = makeKernel(gpu, n);
  const t0 = performance.now();
  const out = k(new Array(n).fill(1), trips);
  const ms = (performance.now() - t0).toFixed(1);
  let gl = null;
  try {
    const c = gpu.canvas;
    if (c && c.getContext) gl = c.getContext('webgl2') || c.getContext('webgl');
    if (gl && typeof gl.getError !== 'function') gl = null;
  } catch (e) { gl = null; }
  const row = {
    mode, n, trips, ms,
    sample: out[0],
    zeroCells: [...out].filter(v => v === 0).length,
    glError: gl ? gl.getError() : 'n/a',
    contextLost: gl ? gl.isContextLost() : 'n/a',
  };
  gpu.destroy();
  return row;
}

const lines = [];
const log = s => { lines.push(s); document.getElementById('out').textContent = lines.join('\n'); };
const gl0 = document.createElement('canvas').getContext('webgl2');
const dbg = gl0.getExtension('WEBGL_debug_renderer_info');
log('renderer: ' + (dbg ? gl0.getParameter(dbg.UNMASKED_RENDERER_WEBGL) : gl0.getParameter(gl0.RENDERER)));
log('');
log('mode  threads  trips        ms      sample        zeroCells  glError  ctxLost');
for (const [mode, n, trips] of [
  ['cpu', 1, 8000000], ['gpu', 1, 1000000], ['gpu', 1, 2000000], ['gpu', 1, 4000000],
  ['gpu', 1, 8000000], ['gpu', 1, 16000000],
  ['gpu', 4096, 4000000], ['gpu', 4096, 8000000], ['gpu', 4096, 16000000],
]) {
  const r = trial(mode, n, trips);
  const want = 1 + trips * 1e-7;
  const flag = r.zeroCells > 0 ? '   <-- WRONG, silently (want ' + want.toFixed(4) + ')' : '';
  log(`${r.mode.padEnd(5)} ${String(r.n).padEnd(8)} ${String(r.trips).padEnd(12)} ${String(r.ms).padEnd(7)} ` +
      `${String(r.sample).padEnd(13)} ${String(r.zeroCells + '/' + r.n).padEnd(10)} ${String(r.glError).padEnd(8)} ${r.contextLost}${flag}`);
}
log('');
log('done');
</script>

Observed

renderer: ANGLE (Apple, ANGLE Metal Renderer: Apple M1 Max, Unspecified Version)

mode  threads  trips        ms      sample        zeroCells  glError  ctxLost
cpu   1        8000000      14.3    1.7999999523162842 0/1        n/a      n/a
gpu   1        1000000      125.3   1.1192092895507812 0/1        0        false
gpu   1        2000000      68.7    1.2384185791015625 0/1        0        false
gpu   1        4000000      128.0   1.476837158203125 0/1        0        false
gpu   1        8000000      133.7   0             1/1        0        false   <-- WRONG, silently (want 1.8000)
gpu   1        16000000     133.3   0             1/1        0        false   <-- WRONG, silently (want 2.6000)
gpu   4096     4000000      94.2    0             4096/4096  0        false   <-- WRONG, silently (want 1.4000)
gpu   4096     8000000      5.9     0             4096/4096  0        false   <-- WRONG, silently (want 1.8000)
gpu   4096     16000000     5.8     0             4096/4096  0        false   <-- WRONG, silently (want 2.6000)

Note the timings on the failing rows — 5.9 ms for work that takes ~130 ms when it succeeds. The draw is being abandoned, not run to completion and then mis-read.

It is not the loop bound or shader compilation

The obvious suspect is loopMaxIterations producing a shader the compiler chokes on. It isn't. A kernel compiled with loopMaxIterations: 40000000 that breaks after 1,000 iterations returns the correct answer in 8.6 ms — same shader, same bound, early exit. The variable that matters is how much work actually executes.

It is nondeterministic

The threshold moves between runs, which is worth knowing before anyone tries to bisect it. In one run output: [4096] failed at 2,000,000 trips; in another the same configuration was correct at 5,000,000 and a single-thread kernel failed at the same count. So this is not a clean per-thread or total-work limit. Expect flakiness when reproducing.

Environment

  • gpu.js 2.20.0
  • Chrome 150, macOS (Apple M1 Max), ANGLE (Apple, ANGLE Metal Renderer)
  • Reproduces headless and headful, and with WebGL2 selected on a real GPU — not SwiftShader
  • cpu backend is correct throughout, so the kernel and the expected values are not in question

Why it matters

For anything that trusts a kernel result — a test suite, a numerical pipeline, a teaching environment — a silent wrong answer is worse than a thrown error or a hang. A hang is visible and can be watchdogged; this returns quickly with data that looks real.

If gpu.js can detect the condition (a fence/sync object that never signals, a robustness extension such as WEBGL_lose_context/GL_KHR_robustness reporting a reset, or a canary cell in the output that the kernel always writes and the reader checks), throwing or at least warning would let callers react. Even a documented note that long kernels can be abandoned by the platform would help.

What I could not determine

  • The root cause. I could not build a working raw-WebGL2 control (my minimal shader returned zeros even in the regime where gpu.js is correct, so it was measuring my own bug, not the platform). So I cannot say from evidence whether this is reachable at the GL layer or whether gpu.js is in a position to detect it. That question is much better answered by someone who knows the backend.
  • Whether it is macOS/Metal-specific. Only tested there. Windows TDR normally produces a detectable device-lost event instead, so the behaviour may well differ.
  • The exact threshold, given the nondeterminism above.

Found while investigating an unrelated question for a site built on gpu.js; happy to run further experiments on this machine if a specific probe would help.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions