Add synchronous ExternalTexture.CreateForJavaScript and deprecate AddToContextAsync

[bghgary]

Overview

This PR adds a synchronous CreateForJavaScript API to the ExternalTexture plugin (deprecating the async AddToContextAsync), adds thread safety, fixes a numLayers bug, and adds an end-to-end rendering test.

ExternalTexture: Synchronous API

The main change introduces CreateForJavaScript (which synchronously returns a Napi::Value) as the replacement for AddToContextAsync (which returned a Napi::Promise). AddToContextAsync is retained as a [[deprecated]] shim that wraps CreateForJavaScript in an already-resolved Napi::Promise::Deferred, so existing await consumers keep compiling.

Old (async — required promises and a separate JS callback to signal completion):

C++:

std::promise<void> textureCreationSubmitted{};
std::promise<void> textureCreationDone{};

auto externalTexture = std::make_shared<Babylon::Plugins::ExternalTexture>(d3d12Resource);

jsRuntime.Dispatch([&](Napi::Env env) {
    auto jsPromise = externalTexture->AddToContextAsync(env);
    env.Global().Get("setup").As<Napi::Function>().Call({
        jsPromise,
        Napi::Value::From(env, width),
        Napi::Value::From(env, height),
        Napi::Function::New(env, [&](const Napi::CallbackInfo&) {
            textureCreationDone.set_value();
        })
    });
    textureCreationSubmitted.set_value();
});

textureCreationSubmitted.get_future().get();
textureCreationDone.get_future().get();

JS:

function setup(externalTexturePromise, width, height, textureCreatedCallback) {
    externalTexturePromise.then((externalTexture) => {
        const outputTexture = engine.wrapNativeTexture(externalTexture);
        scene.activeCamera.outputRenderTarget = new BABYLON.RenderTargetTexture(
            "rt", { width, height }, scene,
            { colorAttachment: outputTexture, generateDepthBuffer: true, generateStencilBuffer: true }
        );
        textureCreatedCallback();
    });
}

New (sync — no promises, no callbacks):

C++:

Babylon::Plugins::ExternalTexture externalTexture{d3d12Resource};

jsRuntime.Dispatch([externalTexture = std::move(externalTexture)](Napi::Env env) {
    auto jsTexture = externalTexture.CreateForJavaScript(env);
    env.Global().Get("setup").As<Napi::Function>().Call({
        jsTexture,
        Napi::Value::From(env, width),
        Napi::Value::From(env, height),
    });
});

JS:

function setup(externalTexture, width, height) {
    const outputTexture = engine.wrapNativeTexture(externalTexture);
    scene.activeCamera.outputRenderTarget = new BABYLON.RenderTargetTexture(
        "rt", { width, height }, scene,
        { colorAttachment: outputTexture, generateDepthBuffer: true, generateStencilBuffer: true }
    );
}

Internally, CreateForJavaScript now calls bgfx::createTexture2D(..., _external) directly in a single call — the previous two-step createTexture2D + bgfx::overrideInternal dance (and its companion "pump an extra frame" in consumers) is gone. The direct path was originally blocked by a WARP E_INVALIDARG on CreateShaderResourceView; a recent bgfx update (#1669) fixes it, so the workaround is no longer needed.

Other ExternalTexture Changes

• Fix numLayers bug: CreateForJavaScript passed a hardcoded 1 for numLayers in Attach(). Changed to m_impl->NumLayers() so texture array metadata (and SRV ArraySize) is preserved.
• Thread safety: Added std::scoped_lock to Width(), Height(), Get(), CreateForJavaScript(), and Update().
• Simplified Update API: Removed layerIndex parameter — the full texture (all layers) is always updated.
• Texture lifecycle: Replaced handle-based tracking with texture-object tracking (AddTexture/RemoveTexture/UpdateTextures).

New: External Texture Array Render Test

• Tests.ExternalTexture.Render.cpp: Creates a 3-slice texture array (R/G/B), renders each slice via a sampler2DArray shader to an external render target, verifies pixels.
• tests.externalTexture.render.ts: JS test with GLSL shaders sampling from a texture array.
• RenderDoc.h/cpp: Optional GPU capture support (disabled by default; enable by defining RENDERDOC).
• Platform Utils: CreateTestTextureArrayWithData, CreateRenderTargetTexture, ReadBackRenderTarget (D3D11, Metal; stubs for D3D12/OpenGL).
• Assertions are strict: color tolerance 2/255 and exact 100% pixel match on all 3 slices.
• Skipped on D3D12 (GRAPHICS_API STREQUAL "D3D12" compile-time gate); D3D12 support is being added separately in Enable Unit Tests on Win32 D3D12 CI #1671.

Consumer Updates

• Apps/PrecompiledShaderTest, Apps/HeadlessScreenshotApp, Apps/StyleTransferApp: migrated to the sync API; removed the "pump an extra frame so overrideInternal applies" blocks.

Testing

• All UnitTests pass on Win32 (RelWithDebInfo), including the new ExternalTexture.RenderTextureArray test (4096/4096 pixels match on all 3 slices).
• PrecompiledShaderTest: 0/1,048,576 pixels differ vs reference.

[Copilot]

Pull request overview

This PR modernizes the ExternalTexture plugin by switching its JavaScript exposure from an async/promise-based flow to a synchronous API, fixing array-layer metadata propagation, and adding an end-to-end texture-array render validation test in UnitTests.

Changes:

• Replace AddToContextAsync (promise-based) with synchronous CreateForJavaScript, and update consumers/tests/docs accordingly.
• Fix the numLayers propagation bug (texture arrays now preserve layer count through attach/update) and add basic thread-safety via a shared mutex.
• Add a new external texture array render test (C++ + JS harness), plus platform helper utilities and optional RenderDoc capture hooks.

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
Plugins/ExternalTexture/Source/ExternalTexture_Shared.h	Implements the synchronous CreateForJavaScript, adds locking in getters/update, and switches tracking to texture objects.
Plugins/ExternalTexture/Source/ExternalTexture_Base.h	Replaces handle tracking with Graphics::Texture* tracking and updates all attached textures on Update().
Plugins/ExternalTexture/Source/ExternalTexture_D3D11.cpp	Updates platform impl signature to match new Update API.
Plugins/ExternalTexture/Source/ExternalTexture_D3D12.cpp	Updates platform impl signature to match new Update API.
Plugins/ExternalTexture/Source/ExternalTexture_Metal.cpp	Updates platform impl signature to match new Update API.
Plugins/ExternalTexture/Source/ExternalTexture_OpenGL.cpp	Updates platform impl signature to match new Update API.
Plugins/ExternalTexture/Include/Babylon/Plugins/ExternalTexture.h	Public API change: exposes CreateForJavaScript and updates threading contract.
Plugins/ExternalTexture/CMakeLists.txt	Removes dependency on JsRuntimeInternal now that texture creation is synchronous.
Plugins/ExternalTexture/Readme.md	Updates documentation sample code from async/promise flow to synchronous creation.
Install/Test/CMakeLists.txt	Removes the old D3D11-specific external texture test source from install tests.
Core/Graphics/Source/BgfxCallback.cpp	Ensures bgfx fatal errors are always printed to stderr (better CI visibility).
Apps/UnitTests/CMakeLists.txt	Adds render test sources/assets; introduces BABYLON_NATIVE_SKIP_RENDER_TESTS option.
Apps/UnitTests/Source/Tests.ExternalTexture.cpp	Updates unit test to validate synchronous CreateForJavaScript.
Apps/UnitTests/Source/Tests.ExternalTexture.Render.cpp	Adds new end-to-end rendering validation for texture arrays rendered into an external RT.
Apps/UnitTests/Source/Tests.ExternalTexture.D3D11.cpp	Removes legacy D3D11-only async/layer-index external texture test.
Apps/UnitTests/Source/Utils.h	Adds new cross-backend helpers for texture arrays, RT creation, and readback.
Apps/UnitTests/Source/Utils.D3D11.cpp	Implements texture-array creation, RT creation, and readback for D3D11.
Apps/UnitTests/Source/Utils.D3D12.cpp	Adds stubs for new helpers (currently not implemented for D3D12).
Apps/UnitTests/Source/Utils.Metal.mm	Implements texture-array creation, RT creation, and readback for Metal.
Apps/UnitTests/Source/Utils.OpenGL.cpp	Adds stubs for new helpers (currently not implemented for OpenGL).
Apps/UnitTests/Source/RenderDoc.h	Declares optional RenderDoc capture helpers for UnitTests.
Apps/UnitTests/Source/RenderDoc.cpp	Implements optional RenderDoc integration (guarded by RENDERDOC).
Apps/UnitTests/JavaScript/webpack.config.js	Adds the new external texture render-test bundle entry.
Apps/UnitTests/JavaScript/src/tests.externalTexture.render.ts	Adds JS-side shader/material setup and slice rendering for the render test.
Apps/UnitTests/JavaScript/dist/tests.externalTexture.render.js	Adds the built JS bundle consumed by UnitTests.
Apps/StyleTransferApp/Win32/App.cpp	Migrates consumer from async external texture creation to synchronous API.
Apps/PrecompiledShaderTest/Source/App.cpp	Migrates consumer from async external texture creation to synchronous API.
Apps/HeadlessScreenshotApp/Win32/App.cpp	Migrates consumer from async external texture creation to synchronous API.

[bghgary]

[Responded by Copilot on behalf of @bghgary]

Hit a deadlock in ExternalTexture::CreateForJavaScript when integrating this PR into a downstream consumer that creates external textures at frame rate. Posting in case it's useful before merge.

Symptom

std::system_error("resource_deadlock_would_occur") thrown from std::_Mutex_base::lock() → unhandled → std::terminate → fast-fail.

Stack (key frames, top → bottom)

std::_Mutex_base::lock                                               <-- throws
ExternalTexture::CreateForJavaScript::<lambda>                       (ExternalTexture_Shared.h, finalizer @ line 100)
Napi::details::FinalizeData<...>::Wrapper                            (napi-inl.h)
chakra!Js::JsrtExternalArrayBuffer::Finalize
chakra!Memory::SmallHeapBlockT<...>::SweepObjects
chakra!Memory::HeapInfo::Finalize
chakra!Memory::Recycler::Sweep
chakra!Memory::Recycler::FinishConcurrentCollect
chakra!ThreadContext::ExecuteRecyclerCollectionFunction
chakra!Js::EnterScriptObject::EnterScriptObject                      <-- GC runs on script entry
chakra!ContextAPIWrapper_Core
chakra!JsGetProperty
napi_get_named_property
Napi::Object::Get
Babylon::JsRuntime::NativeObject::GetFromJavaScript
Babylon::Graphics::DeviceImpl::GetFromJavaScript
Babylon::Graphics::DeviceContext::GetFromJavaScript
ExternalTexture::CreateForJavaScript                                 (ExternalTexture_Shared.h, lock holder @ line 69)

Root cause

CreateForJavaScript takes std::scoped_lock{m_impl->Mutex()} and then, while still holding the lock, calls Graphics::DeviceContext::GetFromJavaScript(env). That call dispatches into the JS engine via napi_get_named_property, which Chakra treats as a script-entry boundary and may run a GC pass.

If the GC sweeps a previously-created Napi::Pointer<Texture> whose ExternalTexture shares the same m_impl (e.g. a texture from an earlier CreateForJavaScript on the same instance), the finalizer lambda also takes std::scoped_lock{impl->Mutex()} on the same thread. std::mutex is non-recursive, so MSVC's STL detects the recursive acquisition and throws.

This is a timing-dependent issue: it requires GC to fire inside the napi_get_named_property call on a thread that already holds m_impl->Mutex() and that holds a finalizable Napi::Pointer to a sibling texture. Single-shot tests that create one texture and stop won't hit it; rapid recreates at frame rate (e.g. video-frame texture rebinds) do.

Suggested fix

Move DeviceContext::GetFromJavaScript(env) out of the lock scope. The DeviceContext lookup is process-scoped and doesn't depend on m_impl's state, so there's no reason for it to be inside the mutex.

 Napi::Value ExternalTexture::CreateForJavaScript(Napi::Env env) const
 {
-    std::scoped_lock lock{m_impl->Mutex()};
-
     Graphics::DeviceContext& context = Graphics::DeviceContext::GetFromJavaScript(env);

+    std::scoped_lock lock{m_impl->Mutex()};
+
     bgfx::TextureHandle handle = bgfx::createTexture2D(...);
     ...
 }

This was the minimal fix that unblocked the deadlocking scenario in our integration. The remaining lock scope still covers all m_impl accesses (Width(), Height(), HasMips(), NumLayers(), Format(), Flags(), Get(), AddTexture()).

A more general guideline that may be worth applying across the file: any path that calls into the JS engine while holding m_impl->Mutex() is at risk of the same recursive-lock pattern, since GC-driven finalizers also take that mutex.

[bghgary]

[Responded by Copilot on behalf of @bghgary]

While reviewing this PR's ExternalTexture implementation in the context of similar work in another project, I encountered a recursive-mutex deadlock on Windows that appears to be a real bug worth filing here.
...

Napi::Value ExternalTexture::CreateForJavaScript(Napi::Env env) const
{
    std::scoped_lock lock{m_impl->Mutex()};
    Graphics::DeviceContext& context = Graphics::DeviceContext::GetFromJavaScript(env);
    ...

Thanks — confirmed and fixed in 415f193. The diagnosis above is correct: Graphics::DeviceContext::GetFromJavaScript(env) performs a JS-side property lookup (_native._Graphics) that can run engine GC and finalizers on Chakra. When a sibling Texture's Napi::Pointer finalizer runs there, it re-enters m_impl->Mutex() on the same thread; on MSVC that recursive lock throws std::system_error("resource_deadlock_would_occur"), which then escapes the AppRuntime::Dispatch lambda and triggers std::abort (the dispatcher only catches Napi::Error specially; anything else hits assert(false); std::abort();).

Fix (415f193): resolve DeviceContext via the JS lookup before acquiring the impl mutex. DeviceContext is process-scoped, so it doesn't need the impl mutex; the lock still covers all m_impl reads and AddTexture.

Regression test (dd75619): ExternalTexture.CreateForJavaScriptDoesNotHoldImplMutexAcrossJsCallout reproduces the deadlock deterministically without needing any Babylon scene, GC pressure, or external scheduling. It redefines _native._Graphics as a Napi accessor whose getter asks the same ExternalTexture for its Width() (which also takes m_impl->Mutex()) and observes whether the lock is currently held on this thread. With the bug present, Width() throws system_error and the test asserts; with the fix, the lookup runs before the mutex is taken and Width() succeeds. Verified to fail on the prior tip and pass on 415f193. Win32-only because the test relies on MSVC's std::mutex deadlock detection to surface the recursive lock as a recoverable exception.

Compare: https://github.com/BabylonJS/BabylonNative/compare/e453e9de..415f193f