Correct gl_FragCoord Y orientation on D3D, Metal and Vulkan

[bkaradzic-microsoft]

Babylon Native's shader model is "shader-visible coordinates are GL-logical, converted to physical at each sampler access". FlipSamplerCoordinatesTraverser (texture v -> 1-v, texelFetch y -> h-1-y) and InvertYDerivativeOperandsTraverser (negate dFdy) implement that for DXBC/DXIL/Metal/Vulkan, and are skipped for OpenGL.

gl_FragCoord was the one shader input still left in physical space. D3D, Metal and Vulkan rasterize with a top-left origin while GL uses bottom-left, and Babylon Native does not flip geometry (ProcessShaderCoordinates only remaps depth), so for GL row y the hardware yields height - y - 0.5 instead of y + 0.5. gl_FragCoord.y arrives mirrored.

Shaders that sample at their own position are unaffected, because the physical/physical pairing is self-consistent. The mismatch only shows up where the row index itself is meaningful: prefix sums (iblCdfy), neighbour offsets, and copies into a differently-oriented target (copyTexture3DLayerToTexture). That is why 39 shaders reference gl_FragCoord but only a handful render incorrectly.

Change

FragCoordYFlipTraverser rewrites every gl_FragCoord read in the fragment stage to vec4(fc.x, targetHeight - fc.y, fc.z, fc.w). The correction is exactly height - y, with no -1 term. Shaders that never read gl_FragCoord are left byte-for-byte unchanged.

The height comes from a new vec4 uniform, bnFragCoordTargetSize, declared as a linker object so MoveNonSamplerUniformsIntoStruct sweeps it into the Frame struct like every other uniform. NativeEngine sets it in DrawInternal from the bound framebuffer's dimensions.

Notes:

• bgfx's predefined u_viewRect is deliberately not reused: FrameBuffer::SetBgfxViewPortAndScissor narrows it to the viewport whenever one is set, whereas gl_FragCoord is relative to the whole render target.
• FlipFragCoordY must run before ChangeUniformTypes / MoveNonSamplerUniformsIntoStruct so its uniform is collected with the rest.
• A fresh replacement subtree is built per occurrence rather than reusing MakeReplacements, which maps one node per symbol name and would give that node multiple parents.
• OpenGL is left alone, as with the other flip traversers.

Tests

Two render-and-readback tests in UnitTests, gated off where the existing render tests are (D3D12, noop Metal device):

• FragCoordYIncreasesUpwards writes gl_FragCoord.y / height into a render target and checks the ramp is brightest at the top row. On a 64-row target it reads 253 / 126 / 2 for the top, middle and bottom rows, exactly the (height - row - 0.5) / height ramp the correction is derived from.
• FragCoordAndUVAddressATextureIdentically samples one texture twice, once through the interpolated UVs of a full-screen quad and once through gl_FragCoord.xy / targetSize, and requires the two images to match. This is the addressing pattern used by order-independent transparency, TAA and screen space curvature, and it holds only if the gl_FragCoord correction and FlipSamplerCoordinatesTraverser compose to a no-op.

Both fail without the fix: the first ramp inverts to 2 / 129 / 253, and the second renders vertically mirrored (255..3 against 3..255).

Validation

• Full UnitTests suite on D3D11. The only failure is the pre-existing JavaScript.All TextEncoder assertion, which is unrelated and also fails without this change.
• 149 Playground validation tests on D3D11 with no pixel-diff regressions.
• IBL voxel shadowing improves measurably: IBL Voxel Shadowing Right-Handed 13.455% -> 13.392% and Left-Handed 14.995% -> 14.592% pixel difference.

Risk

The blast radius is every shader that reads gl_FragCoord on D3D/Metal/Vulkan, so this is worth close review even though the shipped shaders that change behaviour are few.

Only DXBC was exercised on hardware. The DXIL, Metal and Vulkan call sites are the same one-line addition in the same position, but they are untested and would benefit from a run on those backends before merging.

No Playground validation test flips from failing to passing here. The tests that would exercise this most directly (order-independent transparency, TAA, screen space curvature, IBL voxel shadowing) are currently excluded for unrelated reasons, and the voxel tests additionally need a newer babylonjs than the pinned 9.15.0 to run at all. Hence the unit tests above, which pin the behaviour down independently of the npm version.

[Copilot]

Pull request overview

This PR normalizes gl_FragCoord.y to OpenGL/WebGL’s bottom-left-origin convention on the non-OpenGL backends (D3D, Metal, Vulkan) by injecting an AST rewrite during shader compilation and supplying the render-target dimensions at draw time.

Changes:

• Add a new shader-compiler traverser to rewrite every fragment-stage gl_FragCoord read to a Y-flipped equivalent using a new target-size uniform.
• Populate the injected target-size uniform from the currently bound framebuffer dimensions in NativeEngine::DrawInternal.
• Add two render-and-readback unit tests to pin down gl_FragCoord.y orientation and its composition with sampler coordinate flips.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Plugins/ShaderCompiler/Source/ShaderCompilerVulkan.cpp	Runs FlipFragCoordY in the Vulkan compilation pipeline before uniform transforms.
Plugins/ShaderCompiler/Source/ShaderCompilerMetal.cpp	Runs FlipFragCoordY in the Metal compilation pipeline before uniform transforms.
Plugins/ShaderCompiler/Source/ShaderCompilerDXIL.cpp	Runs FlipFragCoordY in the DXIL compilation pipeline before uniform transforms.
Plugins/ShaderCompiler/Source/ShaderCompilerDXBC.cpp	Runs FlipFragCoordY in the DXBC compilation pipeline before uniform transforms.
Plugins/ShaderCompiler/Source/ShaderCompilerTraversers.h	Declares and documents the new FlipFragCoordY traverser API.
Plugins/ShaderCompiler/Source/ShaderCompilerTraversers.cpp	Implements FragCoordYFlipTraverser, declares bnFragCoordTargetSize, and rewrites gl_FragCoord reads.
Core/Graphics/InternalInclude/Babylon/Graphics/BgfxShaderInfo.h	Introduces FRAGCOORD_TARGET_SIZE_UNIFORM_NAME constant for the injected uniform name.
Plugins/NativeEngine/Source/Program.h	Adds cached lookup accessor for the injected uniform’s UniformInfo.
Plugins/NativeEngine/Source/Program.cpp	Caches bnFragCoordTargetSize uniform info during program initialization.
Plugins/NativeEngine/Source/NativeEngine.cpp	Sets bnFragCoordTargetSize each draw based on the bound framebuffer size (when present).
Apps/UnitTests/Source/Tests.ShaderCompilation.FragCoord.cpp	Adds render/readback tests validating gl_FragCoord.y orientation and UV-vs-fragcoord addressing equivalence.
Apps/UnitTests/CMakeLists.txt	Adds the new test source file to the UnitTests build.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[bkaradzic-microsoft]

CI caught a real problem with the test, though not with the fix. Pushed 666e1ad.

What failed: ShaderCompilation.FragCoordYIncreasesUpwards failed on the four Ubuntu jobs (OpenGL) and passed everywhere else. Linux read back 2 / 129 / 253 where Windows D3D11 read back 253 / 126 / 2 — an exact mirror.

Why: the test, not the traverser. Helpers::ReadPixels is a bare glReadPixels(0, 0, ...) on OpenGL, which returns the bottom scanline first, while the D3D11 path maps the staging texture and returns the top scanline first. My test asserted an absolute ramp direction over readback rows, so it silently encoded the D3D11 readback convention. On OpenGL 2 / 129 / 253 is the correct result: readback row 0 is the bottom scanline, which legitimately holds the smallest gl_FragCoord.y.

Worth stating plainly: gl_FragCoord behaviour on OpenGL is unchanged by this PR — FlipFragCoordY is only invoked from the DXBC, DXIL, Metal and Vulkan compilers. The GL result was correct before and after.

Fix: the test now writes normalized gl_FragCoord.y and the interpolated vUV.y into two channels of the same fragment invocation and requires them to agree row by row. The quad maps uv.y = 0 to clip y = -1 and uv.y = 1 to clip y = +1, so the two ramps must co-vary on every backend no matter which end of the image the readback starts at. Renamed to FragCoordYMatchesInterpolatedUV. This is the same orientation-agnostic approach as the second test, which is why that one passed on Linux unmodified.

Re-verified the negative control — with the FlipFragCoordY call commented out on D3D11:

row 0 fragCoord=2 uv=253, row 32 fragCoord=129 uv=126, row 63 fragCoord=253 uv=2
gl_FragCoord.y disagrees with the interpolated vUV.y at row 0 (gl_FragCoord=2, vUV=253)
[  FAILED  ] ShaderCompilation.FragCoordYMatchesInterpolatedUV
[  FAILED  ] ShaderCompilation.FragCoordAndUVAddressATextureIdentically

and with it restored:

row 0 fragCoord=253 uv=253, row 32 fragCoord=126 uv=126, row 63 fragCoord=2 uv=2
[  PASSED  ] 5 tests.

So the reworked assertion still fails by the full range of the ramp when the fix is absent — it did not become weaker by becoming portable.

[bkaradzic-microsoft]

Short answer: no — and I can now show why rather than just reporting that the candidates I tried didn't flip.

I instrumented FragCoordYFlipTraverser with two counters: one that fires every time the traverser scans a fragment stage, and one that fires only when it finds at least one gl_FragCoord read to rewrite. Then I ran the Playground suite under the D3D11 backend.

Result: the traverser is invoked constantly and matches nothing. Across a 37-test spread sampling the whole 720-test catalog, plus ~40 hand-picked tests covering every feature whose shaders reference gl_FragCoord (OIT, TAA, screen-space curvature, volumetric lighting, clustered lighting, fluid rendering, prepass SSAO, IBL voxel/screen-space shadowing, GPU picking, shadow maps, Gaussian splatting):

fragment stages scanned : ~95
stages containing a gl_FragCoord read : 0

The reason is that in babylonjs 9.15.0 — the version pinned by Apps/package-lock.json, which is what npm install resolves for CI — essentially every gl_FragCoord use is behind a define that these tests don't enable:

Shader	Guard
shadowMapPixelShader	#if SM_SOFTTRANSPARENTSHADOW==1
copyTextureToTexturePixelShader	#ifdef NO_SAMPLER
geometryPixelShader	#ifdef IBL_SHADOW_TEXTURE

and the features that do use it unconditionally — OIT, TAA, curvature, volumetric, clustered lighting, the FrameGraph tests — abort before any shader is compiled, e.g.:

Failed to evaluate playground snippet #SYQW69#1366:
TypeError: Cannot read properties of undefined (reading 'COLOR_ATTACHMENT0')

Those depend on engine APIs added to Babylon.js after 9.15.0, so they fail during scene construction regardless of this change.

Two consequences worth stating explicitly:

1. No validation test can be enabled by this PR on the pinned Babylon.js version. Enabling any of the candidates would produce a red CI for reasons unrelated to gl_FragCoord.
2. This PR provably cannot regress the validation suite either — the rewrite is inert for every shader the suite currently compiles. That is consistent with CI, where the only failure was my own unit test making a bad assumption about readback row order.

This is precisely why the change ships with the two GPU-readback unit tests: they exercise gl_FragCoord directly and are independent of which Babylon.js version is pinned. Both have verified negative controls — comment out the FlipFragCoordY call and both fail by the full range of the ramp.

Once the pinned Babylon.js moves past 9.15.0 the OIT / TAA / curvature / clustered-lighting tests become the natural integration coverage for this, and I'm happy to follow up with a PR enabling them at that point.

[bkaradzic-microsoft]

Pushed three follow-up commits: the Babylon.js 9.21.2 bump, an engine fix the bump exposed, and the validation tests it unlocks.

The bump exposed a real bug

VertexArray::RecordVertexBuffer rejected more than 5 per-instance vertex attributes, citing "bgfx allows instancing on at most 4 vec4 attributes". That limit is stale — bgfx reports its capacity in caps.limits.maxInstanceData, which is 16 in the pinned BabylonJS/bgfx (bkaradzic/bgfx a4555482f, "Increased number of texcoord attributes, increased instance data"). I confirmed the pinned bgfx.cmake revision in this branch (865b8e6a) contains that commit, so the higher limit is available on CI and not just locally.

Babylon.js 9.21 draws thin instances with render-self motion blur using world0-3 plus previousWorld0-3 — 8 instanced attributes — so the stale cap threw out of _renderWithThinInstances and the mesh silently failed to draw. Reading the limit from bgfx fixes tests 321/322/323, which regress on 9.21.2 without it.

Regression check

All 302 previously-enabled tests, one process per test, on Win32 D3D11: 299 pass, 0 fail. The only non-passing are 53-55 (scissor), which crash identically on 9.15.0 — pre-existing on Windows/D3D11 and unrelated to this change.

9 tests enabled

Several of these exclusions describe order-dependent behaviour, which a per-test sweep structurally cannot reproduce, so I also ran a single sequential process over indices 56-719: ran=256 passed=256 failed=0.

#	Test	Diff
137	Volumetric Light Scattering + Morph Targets	exact
287	Prepass SSAO + particles	exact
289	Prepass SSAO + instanced bones	0.012%
299	Prepass SSAO + GUI	1.064%
302	Prepass SSAO + highlight layer	0.018%
304	Prepass SSAO + on/off post-process	exact
305	Prepass SSAO + thin instances	0.003%
306	Prepass SSAO + depth renderer	0.044%
363	Screen Space Reflections 2	1.315%

Two things I want to flag honestly

1. Three of these can only be judged by CI. Their exclusion reasons are backend-specific and I have no way to reproduce them on a D3D11 host: 137 "fails on Linux (large diff)", 287 "fails to compile on desktop GL", and 299 OpenGL mediump float compile failure. If the Ubuntu jobs go red on any of these, the right response is to re-exclude that test with an updated reason rather than to paper over it.

2. Tests 321 and 323 are now marginal — 99.1% and 94.2% of their error budget (2.478% and 2.355% against 2.5%). Bit-identical across three runs, so not flaky, but that margin is unlikely to survive a different backend. The cause is visible in the render: Babylon Native leaves a soft motion-blur halo around objects where Babylon.js converges to zero velocity, so there is a residual gap in the motion-blur path beyond the instance-limit bug. I ruled out stale reference images (substituting Babylon.js's own PNGs gives identical diffs). Worth a follow-up issue; I did not want to hide it behind a raised errorRatio.

Note that none of the newly enabled tests exercise gl_FragCoord — they are unlocked by the bump, not by the traverser. The tests that do prove the traverser are the clustered-lighting ones, and those additionally need StorageBuffer support that is not in this branch. The unit tests remain the proof for the gl_FragCoord change itself.

[bkaradzic-microsoft]

Correction to my previous comment: the instance-limit commit I pushed here duplicated work that already exists in #1839, so I have dropped it and force-pushed. VertexArray.cpp is now untouched by this PR.

This PR now depends on #1839. The Babylon.js 9.21.2 bump regresses three tests that #1839 fixes:

Test	with bump, without #1839	with #1839
321 Thin instances + dynamic buffer resize	3.346%	2.478%
322 Instances + render self motion blur	2.531%	2.061%
323 Thin instances + render self motion blur	2.844%	2.355%

Until #1839 merges, those three will fail here. They should not be worked around in this PR.

For the record, my dropped commit and #1839 converged on the same guard independently (find(attrib) == end() && size() >= max, replacing the stale size() > 4), which is a useful cross-check on that reasoning. But #1839 is the correct fix and mine was not: it also maps previousWorld0-3 with dense per-shader slot assignment, which is the actual root cause. My limit-only change happened to reach the same three numbers via the existing generic-attribute reroute, but it left the contiguity guarantee resting on luck rather than construction.

#1839 also explains the residual halo I flagged, and it is a third, separate bug: bindAttachments is a no-op on Native, so the scene clear is applied to every MRT attachment and wipes the velocity attachment's alpha-0 background, giving the background a fixed non-zero velocity. That is why 321 still sits at 99.1% of its error budget even after #1839, and it needs a matching Babylon.js change to fix properly.

The rest of this PR is unchanged: the gl_FragCoord traverser plus its unit tests, the 9.21.2 bump, and the 9 tests the bump unlocks.