[wasm] Fix JIT helper calls modeled with the wrong return arity

[lewing]

Summary

Three JIT helper calls on wasm were modeled with a return arity that does not match the helper they call. On wasm the emitted call signature is derived from the JIT-modeled return type in CodeGen::genCallInstruction:

const var_types callRetType = call->IsFastTailCall() ? (var_types)call->gtReturnType : genActualType(call);
if (call->ShouldHaveRetBufArg() || (callRetType == TYP_VOID)) { typeStack.Push(CORINFO_WASM_TYPE_VOID); }

so the common "model a value-returning helper as TYP_VOID because the result is unused" shortcut — harmless on register machines — produces a function signature mismatch engine trap. This is the same class of defect as #130813, which fixed it for the class-init helpers.

1. Inline-unbox slow path (importer.cpp)

The inline expansion of unbox lowers to (*clone == typeToken) ? nop : CORINFO_HELP_UNBOX(clone, typeToken) and created the slow-path helper call as TYP_VOID, because the unboxed byref is formed separately from clone + TARGET_POINTER_SIZE. But CastHelpers.Unbox returns ref byte, so this emitted a (...)->() call_indirect for a helper the runtime dispatches through an i32-returning thunk — e.g. on the first cold unbox in System.Enum.ToObject during reflection-heavy startup.

Follows #130813's shape with a HelperUnboxRetType constant, and discards the unused result via gtUnusedValNode so the enclosing COLON/QMARK remain void. On targets where the helper stays TYP_VOID the COMMA folds away in morph. The non-inline path in the same switch already modeled this helper as TYP_BYREF, so the two paths are now consistent.

2. ObjectAllocator::RewriteUses (objectalloc.cpp) — fixes #131377

This pass retargets a CORINFO_HELP_UNBOX call node to CORINFO_HELP_UNBOX_TYPETEST when the unboxed object is a stack-allocated box, but never retyped the node. CastHelpers.Unbox returns ref byte while CastHelpers.Unbox_TypeTest returns void, so the node was left claiming a byref return for a void helper.

isForEffect is still computed from the original type so the existing COMMA(call, ADD(obj, TARGET_POINTER_SIZE)) shape is preserved.

The two commits must ship together. Change 1 is what makes this reachable: previously the inline-unbox call arrived at this pass as TYP_VOID, so isForEffect was true and the void model matched Unbox_TypeTest by accident. This was confirmed experimentally — see validation below.

3. CORINFO_HELP_CHECK_OBJ in fgMorphInit (morph.cpp) — fixes #131378

JIT_CheckObj returns Object*, and importer.cpp already models it as TYP_REF; only fgMorphInit used TYP_VOID. Only fires under DOTNET_JitGCChecks.

This one is defensive — it has no reachable runtime manifestation today, and the issue as I originally filed it overstated the impact. CORINFO_HELP_CHECK_OBJ is not handled anywhere in CorInfoImpl.ReadyToRun.cs, so it falls through to default: throw new NotImplementedException — it is not even in the graceful RequiresRuntimeJitException list. Any method that would emit CHECK_OBJ therefore fails R2R codegen outright, on every target, long before return-arity modeling reaches call_indirect emission:

ILCompiler.CodeGenerationFailedException: Code generation failed for method 'GcCheck.Foo(object)'
 ---> System.NotImplementedException: CORINFO_HELP_CHECK_OBJ
   at CorInfoImpl.GetHelperFtnUncached(...) CorInfoImpl.ReadyToRun.cs:1342

So no R2R image containing a CHECK_OBJ call can be built, and wasm has no runtime JIT (R2R + interpreter only), so there is no other path where the modeling would bite. The change is kept because it aligns the model with JIT_CheckObj's actual Object* return and with the model the importer already uses for the same helper, and it would become live if CHECK_OBJ ever gains R2R support.

Notes for reviewers

• The objectalloc.cpp change is deliberately not wasm-guarded. The node was mistyped on every target; it is simply benign on register machines, where the unused return value is left in the return register. Correcting it there too seemed better than adding a target-specific divergence, but this is the part of the PR with broad blast radius and is the main thing worth a second opinion.
• Changes 1 and 3 leave non-wasm codegen alone: both helpers keep their existing TYP_VOID / effective-void modeling, and the added gtUnusedValNode wrapper collapses back to the bare call in morph when the call is TYP_VOID.

Validation

Verified on a browser R2R rig (there is no wasm-R2R CI leg, so this cannot be validated by CI alone).

#131377 proven as a real runtime trap, statically and dynamically. Disassembling the stack-allocated unbox method shows the defect directly at the UNBOX_TYPETEST site:

	emitted
unfixed	call_indirect (type 5) + drop — i32-returning signature for a void helper
fixed	call_indirect (type 4) — void, no drop

The images differ only at that site. Runtime differential across three JIT states, forced through the helper:

JIT state	result
change 1 only (no objectalloc fix)	function signature mismatch trap, ec=1, 3/3
both changes	helper runs, type-tests, throws InvalidCastException (caught) → ec=100, 3/3
neither change	ec=100, no trap

The third row confirms change 1 alone regresses this path, hence the ship-together requirement.

Regression sweep, both commits combined: System.Text.Json discovery 12166/12166 with sig-mismatch=0, nre=0, oob=0 across seeds 1/42/99; Unsafe 128/128 and Bcl.Memory 550/550 with both R2R-off and R2R-on, trap=0. The HelperUnboxRetType + gtUnusedValNode refactor produces behavior identical to the original inline #ifdef form.

Build/format: clr.alljits and the clrjit_universal_wasm cross-JIT build clean in Checked and Release on osx-arm64, 0 warnings; clang-format (repo's own, 17.0.6) reports 0 replacements.

Native (arm64) before/after

I was initially unsure whether the un-guarded objectalloc.cpp change affects native codegen at all. It does, on one narrow path, and the effect is measurable.

On the hot path the node already arrives as TYP_VOID on native (the importer's inline-unbox expansion), so isForEffect is true and the retype is a no-op — fixed and unfixed JITs produce identical trees. But when the unbox lands in a rarely-run block the importer takes the non-inline path, which models the helper TYP_BYREF, and the bug reproduces on arm64:

JIT	node
unfixed	CALL help byref CORINFO_HELP_UNBOX_TYPETEST
fixed	CALL help void CORINFO_HELP_UNBOX_TYPETEST

Diffing the full arm64 dumps for that method, with ASLR address noise filtered out, the only differences are:

• the call node's type and register def (REG x0 → REG NA), and
• two removed GC-tracking lines, Byref regs: 0000 {} => 0001 {x0} / Byref regs: 0001 {x0} => 0000 {}.

Total bytes of code 236, instruction count 59 is identical in both. So the unfixed JIT was briefly reporting x0 as a live byref in GC info across a call to a helper that returns nothing. It is dead immediately, so it is not practically observable, but it is bogus GC info rather than a purely cosmetic modeling wart.

Practical upshot for review: expect GC-info diffs from SPMI, not instruction diffs.

Still outstanding

• No SPMI run. The arm64 before/after above is a single hand-built repro, not a corpus sweep; SPMI should confirm the "GC-info diffs only, no instruction diffs" expectation across real code.
• Change 3 ([wasm] CORINFO_HELP_CHECK_OBJ is modeled TYP_VOID in fgMorphInit but JIT_CheckObj returns Object* #131378) is not runtime-testable by construction, per the crossgen2 NotImplementedException above — not merely untested. Verified by building the Checked wasm JIT and attempting to crossgen a GC-ref-param method with --codegenopt:JitGCChecks=1.
• No jit-format/clang-tidy run.

Regression test

Added src/tests/JIT/Regression/JitBlue/Runtime_131377.

The existing src/tests/JIT/opt/ObjectStackAllocation/ObjectStackAllocationTests.cs Case3 was written to trigger CORINFO_HELP_UNBOX_TYPETEST, but as written it would not catch this trap even under R2R. The call_indirect is guarded by an inline method table check that short-circuits the helper whenever the unbox type matches. Case3 is only ever called with null, which stack-allocates a Guid and takes the matching path, so the mismatched instruction is compiled but never executed. It also cannot simply be extended in place: ZeroAllocTest captures GC.GetAllocatedBytesForCurrentThread() once and asserts zero allocation after each case, so allocating a non-matching object there would break the assertion for every subsequent case. That project additionally sets <CrossGenTest>false</CrossGenTest> (#109314), so it gets no R2R coverage today.

The new test drives a non-matching type through the same unbox site so the type-test slow path actually executes, and sets AlwaysUseCrossGen2 for TargetOS=browser so the browser leg compiles it to wasm R2R instead of running it as ordinary JIT and false-passing.

Verified locally on osx-arm64 Checked that it compiles, that the assertion is correct (InvalidCastException is thrown, exit 100), and — via JIT dump — that it genuinely exercises the path: Allocating V03 on the stack followed by Rewriting to invoke box type test helper.

Confirmed to be a real guard on the browser R2R rig, crossgen'd standalone (--targetos:browser --targetarch:wasm -O) against each JIT and run under WasmTestRunner + node, 3x each:

JIT	result
fixed	Tests run: 1, Passed: 1, Failed: 0, ec=0, sig-mismatch=0 — 3/3
unfixed (objectalloc fix reverted)	function signature mismatch, module aborts before any result, ec=1 — 3/3

On the unfixed JIT the engine traps at the call_indirect before Assert.Throws ever runs, so the test fails as a crash rather than an assertion failure. The two test images differ only at the UNBOX_TYPETEST site.

Follow-up

#131379 tracks the systemic gap: nothing validates modeled helper return arity against the real helper signature, which is why this class of bug has now been found three times by runtime traps rather than at JIT time.

Contributes to #129622.

Note

This PR was created with assistance from GitHub Copilot (AI).

[azure-pipelines]

Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR corrects several CoreCLR JIT helper-call nodes whose modeled return type didn’t match the actual helper signature, which is especially problematic for wasm where the emitted call signature is derived from the modeled return type.

Changes:

• Retypes an UNBOX → UNBOX_TYPETEST helper retarget in ObjectAllocator::RewriteUses to ensure the call node becomes TYP_VOID.
• Updates fgMorphInit GC-check insertion to model CORINFO_HELP_CHECK_OBJ as value-returning and explicitly discard the result.
• Updates the inline unbox slow-path helper call to use a wasm-appropriate modeled return type and discard the result via gtUnusedValNode.
• Introduces a wasm-specific modeled return type constant for the unbox helper to centralize this choice.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/coreclr/jit/objectalloc.cpp	Retypes retargeted UNBOX_TYPETEST helper calls to TYP_VOID to match helper signature.
src/coreclr/jit/morph.cpp	Models CHECK_OBJ as TYP_REF and discards the unused result to keep statements effectively void.
src/coreclr/jit/importer.cpp	Models inline-unbox slow-path helper calls with a wasm-correct return type and discards the value.
src/coreclr/jit/compiler.h	Adds a wasm-specific constant to drive modeled helper return type for discarded unbox results.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.