[PIX] Fix mesh shader instrumentation

[Damyan Pepper (damyanp)]

Part 14 of 14 in the PIX instrumentation stack. It targets users/damyanp/pix-fixes-13. Its content depends on PR 2 for the declaration cleanup and the shared UAV behavior.

PIX instruments an amplification shader by expanding its payload to carry a group identifier. The mesh shader must agree on the resulting layout. When the two cannot be reconciled, the pass applies a locally derived layout that is known to disagree. The instrumented pair then disagrees about where each payload field lives.

A mesh shader whose payload is never read is skipped, so its output cannot be instrumented at all. A signed 16-bit output is zero-extended instead of sign-extended. Unused EmitIndices declarations stay in the module, and the validator refuses it. The reservation assertion checks a value that is already zero, so it does not catch a caller that asks for enough space to overwrite the offset counter.

The amplification shader reports the size and offset of the expanded payload through the expanded-payload-size and expanded-payload-offset options. The mesh shader reconstructs its payload type from that report. When reconstruction fails, the pass reports the failure and leaves the mesh payload unchanged. A mismatched pair produces records PIX cannot interpret, so there is no fallback layout.

The pass does not expand a payload that is already near the size limit.

If the PIX coordinator does not forward the layout of the amplification shader, the mesh shader leaves its payload unchanged. That is an integration item for PIX.

Assisted-by: Copilot

This changes only the PIX instrumentation, so it needs no release note.

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[Copilot]

Pull request overview

Fixes PIX mesh-shader instrumentation so amplification and mesh shaders can share an expanded payload layout.

Changes:

• Reports and consumes expanded payload size and offset.
• Adds payload limits, alignment fixes, signed 16-bit handling, and declaration cleanup.
• Adds regression and validation coverage.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/hct/hctdb.py	Registers payload-layout pass options.
tools/clang/unittests/HLSL/PixTest.cpp	Adds PIX instrumentation regression tests.
tools/clang/test/HLSLFileCheck/pix/SynthesizedPayloadTypeShapes.hlsl	Tests synthesized payload layouts.
tools/clang/test/HLSLFileCheck/pix/SynthesizedPayloadTypeRejectsUnusableLayout.hlsl	Tests invalid-layout rejection.
tools/clang/test/HLSLFileCheck/pix/MeshShaderWithoutEmitIndicesLeavesNoDeadDeclaration.hlsl	Verifies declaration cleanup.
tools/clang/test/HLSLFileCheck/pix/MeshPayloadExpansionAgreesWithAmplificationShader.hlsl	Verifies AS/MS layout agreement.
tools/clang/test/HLSLFileCheck/pix/MeshOutputSignedInt16IsSignExtended.hlsl	Verifies signed 16-bit extension.
tools/clang/test/HLSLFileCheck/pix/AddThreadIdWhenMSPayloadIsUnused.hlsl	Updates unused-payload expectations.
lib/DxilPIXPasses/DxilPIXMeshShaderOutputInstrumentation.cpp	Reconstructs payload layouts and fixes output instrumentation.
lib/DxilPIXPasses/DxilPIXAddTidToAmplificationShaderPayload.cpp	Reports layout and enforces payload constraints.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

The mesh pass still applies a locally derived fallback layout when the coordinator omits the amplification-shader layout.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• When no expanded payload size is forwarded, this branch still derives and installs a local layout.…
• Use explicit StructLayout pointer declaration · New
• Use explicit StructType pointer declarations · New
• Use explicit CComPtr<IDxcBlob> declarations · New

[Copilot]

Copilot review overview

🟡 Changes recommended

The mesh pass still applies a locally derived payload layout when no amplification layout is forwarded.

Review effort: Balanced
Findings: 1 · 1 · 3

Open (5)

• Avoid fallback mesh expansion when forwarded payload layout is missing · New
• When no expanded payload size is forwarded, this branch still derives and installs a local layout.…
• Use explicit CComPtr<IDxcBlob> declarations
• Use explicit StructType pointer declarations
• Use explicit StructLayout pointer declaration

[Copilot]

Copilot review overview

🔵 Needs a closer look

Mesh payloads are still locally expanded when the coordinator omits the amplification-shader layout, contrary to the stated contract.

Review effort: Balanced
Findings: 1 · 1 · 3

Open (5)

• Avoid fallback mesh expansion when forwarded payload layout is missing
• When no expanded payload size is forwarded, this branch still derives and installs a local layout.…
• Use explicit CComPtr<IDxcBlob> declarations
• Use explicit StructType pointer declarations
• Use explicit StructLayout pointer declaration

[Copilot]

Copilot review overview

🟡 Changes recommended

The fallback can still create mismatched layouts, and expansion can exceed the combined mesh input/output limit.

Review effort: Balanced
Findings: 2 · 1 · 3

Open (6)

• Account for mesh output footprint in payload expansion limit · New
• Avoid fallback mesh expansion when forwarded payload layout is missing
• When no expanded payload size is forwarded, this branch still derives and installs a local layout.…
• Use explicit CComPtr<IDxcBlob> declarations
• Use explicit StructType pointer declarations
• Use explicit StructLayout pointer declaration

[Copilot]

Copilot review overview

🟡 Changes recommended

Missing coordinator layout options still trigger the incompatible local payload-layout fallback.

Review effort: Balanced
Findings: 2 · 1 · 4

Open (7)

• Account for mesh output footprint in payload expansion limit
• Avoid fallback mesh expansion when forwarded payload layout is missing
• When no expanded payload size is forwarded, this branch still derives and installs a local layout.…
• Use explicit LLVM Type pointer types instead of auto · New
• Use explicit CComPtr<IDxcBlob> declarations
• Use explicit StructType pointer declarations
• Use explicit StructLayout pointer declaration

[Copilot]

Copilot review overview

🟡 Changes recommended

The mesh pass still applies a local fallback layout when the coordinator provides no amplification layout, contradicting the intended safety behavior.

Review effort: Balanced
Findings: 2 · 1 · 5

Open (8)

• Account for mesh output footprint in payload expansion limit
• Avoid fallback mesh expansion when forwarded payload layout is missing
• When no expanded payload size is forwarded, this branch still derives and installs a local layout.…
• Use explicit const std::string reference type · New
• Use explicit LLVM Type pointer types instead of auto
• Use explicit CComPtr<IDxcBlob> declarations
• Use explicit StructType pointer declarations
• Use explicit StructLayout pointer declaration