Skip to content

[FIX] Run NKI kernels written like the official samples in the NKI interpreter - #539

Open
mark14wu wants to merge 2 commits into
mainfrom
fix/nki-official-kernels
Open

mark14wu wants to merge 2 commits into
mainfrom
fix/nki-official-kernels

Conversation

@mark14wu

@mark14wu mark14wu commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The legacy NKI interpreter (frontend="nki") runs only kernels written in a restricted style. Kernels in the style of AWS's NKI samples fail before their first tile operation. Of seven tutorial kernels from the samples (softmax, rmsnorm, two layernorms, transpose2d, average pool 2-D, SPMD tensor addition), only the SPMD addition ran under the tracer on main. The failures come from a few idioms:

Idiom Failure on main
nl.store(o[i], value=nl.copy(nl.load(x[i]))): a load or store nested in another call NKITransformer rewrote nl.load(x[...]) to nl.load(x, keys) only when the call was a whole statement, so the nested call lost its keys: TypeError: missing ... 'keys'
nl.load(t), nl.store(t, value=v): whole-tensor access no keys at all, same error
nl.load(x[i * 128 + ix, iy], mask=(i * 128 + ix < n)): a [128, 1] row mask on a [128, F] tile NKI broadcasts masks; masked_load asserts that the mask has the tile's shape
nl.load(x[...]) without a mask the tracer read mask.data of None
out = nl.ndarray(..., buffer=nl.shared_hbm) ... return out NKIInterpretedFunction.run dropped the kernel's return value, so the launch returned None
nl.max(t, axis=[1], keepdims=True), nl.sum(t, axis=[1]) numpy rejects a list of axes; and NKI keeps the last free axis: reducing the free axis of a [128, F] tile gives [128, 1], so t / nl.sum(t, axis=[1]) broadcasts in NKI and failed here
nl.max, nl.min, nl.mean, nl.add, nl.subtract, nl.divide, nl.maximum, nl.minimum, nl.static_cast, nl.par_dim, nl.sequential_range, nl.static_range not patched, so the kernel called NKI's real implementation: RuntimeError: calling NKI API outside of NKI kernels is not supported

This PR:

  • NKITransformer.visit_Call rewrites nl.load / nl.store / nl.load_transpose2d with a subscripted first argument wherever the call appears.
  • Load and store keys default to the whole tensor (full_keys), in the interpreter and in the frontend's Load / Store adapters, so clients see the same keys.
  • tile_shape / tile_mask (in core/masked_load_store.py) give the shape of ndarray[keys] even when some lanes are out of bounds, and broadcast an NKI mask to it. The interpreter broadcasts masks and stored values with them; the tracer records the broadcast mask, or an all-true mask for an access without one.
  • The runner returns the last program's return value, converted to numpy arrays. Every program returns the same shared-HBM arrays. It also returns them when a client stops the grid early (second commit); before, the early return dropped them.
  • Reductions go through Builder._reduce: a list of axes becomes a tuple, mask becomes numpy's where (with the dtype's lowest or highest value as the identity of max / min), dtype casts the result, and a result that would lose its last free axis gets [P, 1], as NKI's simulator gives. Element-wise operations take dtype; nl.load takes dtype too.
  • The operations in the last row of the table are implemented and patched. par_dim returns the size, and the three ranges iterate like range.

The shapes and results were checked against NKI's own CPU simulator (neuronxcc.nki.simulate_kernel). For example, nl.sum over axis [1] of an [8, 4] tile gives (8, 1); over axes [1, 2] of an [8, 4, 3] tile, (8, 1) without keepdims and (8, 1, 1) with it. With this PR, six of the seven tutorial kernels run under the tracer. The seventh, layernorm v2, uses nisa.bn_stats, which the interpreter does not implement yet.

Tests

  • New tests/nki/test_nki_kernel_idioms.py: four kernels in the samples' style run under TileLens and under nki.simulate_kernel; their outputs must match. The kernels cover a masked softmax over 200 rows, whole-tensor and nested loads and stores, an SPMD addition on a 2×2 grid, and a kernel with loops, casts, and binary operations. All four fail on main. A fifth test stops the grid after the first program with a client whose post_run_callback returns False and checks that the launch still returns that program's output; it fails without the second commit.
  • tests/nki/test_nki_tracer.py::test_tracer_records_reduce_sum now expects the reduction's output shape (4, 1) instead of (4,), as NKI's simulator gives. Storing that result into the test's 1-D output slice still works: a stored value with the tile's number of elements is reshaped to the tile.
  • pytest tests/ -m "" (NKI tests included): 621 passed, 2 skipped on main; 626 passed, 2 skipped with this PR. Environment: Python 3.12, Triton 3.8.0, neuronx-cc 2.22.12471, nki 0.2.0.
  • ruff check and ruff format --check (v0.1.9) pass on the changed files.

🤖 Generated with Claude Code

…terpreter

The legacy NKI interpreter failed on the idioms of AWS's NKI samples:
- `nl.load(x[...])` / `nl.store(o[...], ...)` nested in another call was not
  rewritten to the (tensor, keys) form, so the call lost its keys;
- `nl.load(t)` and `nl.store(t, value=v)` without an index had no keys at all;
- masks broadcast to the tile in NKI (a [128, 1] row mask on a [128, F]
  tile), but the interpreter and the tracer required the tile's exact shape,
  and the tracer failed on a load without a mask;
- a kernel that allocates its output in shared HBM and returns it returned
  None, because the runner dropped the kernel's return value;
- reductions took a list of axes, which numpy rejects, and dropped the last
  free axis, where NKI keeps [P, 1];
- max, min, mean, add, subtract, divide, maximum, minimum, static_cast,
  par_dim, sequential_range, and static_range were not patched, so they ran
  NKI's real implementation and failed outside an NKI kernel.

Load/store keys default to the whole tensor, masks and stored values are
broadcast to the tile shape (`tile_shape` / `tile_mask` in
masked_load_store.py), the runner returns the kernel's outputs as numpy
arrays, and the missing operations are implemented with NKI's semantics
(dtype casts, masked reductions, the [P, 1] reduction shape).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Performance Benchmark

Benchmark main (min) PR (min) Change Samples
gemm 0.054s 0.055s +1.2% 20 / 20
gemm_oob 0.060s 0.061s +0.8% 20 / 20
indirect_load 0.011s 0.011s -1.8% 20 / 20
nested_loop 0.111s 0.111s +0.7% 20 / 20
block_pointer_loop_advance 0.059s 0.060s +1.6% 20 / 20
liger_jsd 0.076s 0.076s -0.2% 20 / 20
flaggems_layernorm 0.190s 0.199s +4.6% 20 / 20
swiglu 0.092s 0.092s +0.3% 20 / 20
cross_entropy 0.517s 0.517s +0.0% 20 / 20
fused_linear_jsd 0.117s 0.117s -0.6% 20 / 20
Total 1.288s 1.299s +0.8% N/A

Iterations: 1 warmup + 20 measured
Samples are shown as main / PR; long pytest benchmarks may use fewer samples.

A client whose pre_run or post_run callback returns False stops the
remaining programs; the runner then returned None instead of the outputs
the programs so far wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
None stays None."""
if mask is None:
return None
return np.broadcast_to(np.asarray(mask, dtype=bool), tile_shape(ndarray, keys))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tile_mask/tile_shape only clip integer-array keys, so for an out-of-bounds slice key the 'tile shape' is numpy's truncated slice shape, and broadcasting the full-tile mask to it raises. Masked tail loads and stores that use slices now crash, where before this change they worked.

a.shape == (100, 4), nl.load(a[0:128, :], mask=m) with m.shape == (128, 4): tile_shape returns (100, 4) and np.broadcast_to((128,4) -> (100,4)) raises ValueError (I reproduced this). Before, masked_load's padded slow path returned a (128, 4) tile. The same failure hits Builder.store (value and mask are broadcast to (100, 4)) and the tracer's _nki_offsets_and_mask. Fix: compute the shape from slice.stop as _calculate_target_shape does, not from ndarray[keys].

ndarray = dst.data
value_array = value.data
mask_value = getattr(mask, "data", mask) if mask is not None else None
keys = full_keys(dst) if keys is None else keys

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With keys made optional, a whole-tensor store that passes the value positionally (nl.store(out, v), which matches NKI's store(dst, value) signature) binds v to keys, leaving value=None.

nl.store(out_tensor, result_tile): the transformer does not rewrite it (arg0 is not a Subscript), so the call becomes Builder.store(out_tensor, keys=result_tile, value=None). numpy_keys is then a float array, tile_shape indexes with it and raises IndexError. The NKI Store adapter in frontend/nki.py has the same mis-binding, so the tracer crashes first.

value=new_call,
type_comment=getattr(node, "type_comment", None),
)
def visit_Call(self, node: ast.Call) -> ast.AST:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rewrite only looks at a positional first argument. nl.store(dst=o[i], value=v) and nl.load(src=x[i]) keep the subscript, so they act on an NDArray copy made by getitem.

nl.store(dst=out[i*128 + ix, iy], value=t): NDArray.getitem returns a copy, and Builder.store then uses full_keys on that copy. The store goes into a temporary and is lost without any error, so the returned output stays uninitialized.

same_size = value_array.size == int(np.prod(tile_shape_))
if value_array.shape != tile_shape_ and same_size:
# e.g. a [P, 1] reduction stored into a 1-D slice
value_array = value_array.reshape(tile_shape_)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

store reshapes any value whose element count matches the tile, whatever its shape. Real shape mismatches are silently reinterpreted instead of raising.

Storing a (4, 8) tile into an (8, 4) slice (for example a forgotten transpose) passes same_size and is reshaped row-major. Scrambled data is written with no error, while NKI would reject the shape. Limit the reshape to a size-1 axis being dropped or added (e.g. [P,1] -> [P]).

):
key = np.clip(key, 0, max(ndarray.shape[dim] - 1, 0))
clipped.append(key)
return ndarray[tuple(clipped)].shape

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tile_shape does not handle out-of-bounds scalar integer keys. It also lines keys up with array dims by key position, which is wrong once a None (newaxis) key appears.

A masked nl.load(a[r, :], mask=False_mask) with r >= a.shape[0] now raises IndexError in tile_shape, although masked_load's slow path handles int keys via _calculate_target_shape. With keys (None, idx), idx is clipped against ndarray.shape[1] but indexes axis 0. For a.shape == (4, 100) and idx = arange(50), the clip leaves values up to 49 and indexing raises IndexError.

value = np_func(x.data, axis=_axis(axis), keepdims=keepdims, **reduce_kwargs)
if not keepdims and np.ndim(value) == 1 and x.data.ndim >= 2:
# NKI keeps a free dimension: reducing all free axes of a [P, ...] tile gives [P, 1]
value = value[:, None]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule that keeps the free dim adds a trailing axis to any 1-D result from a 2-D or larger input. That includes reducing the partition axis.

nl.sum(x, axis=0) on a (P, F) tile gives a numpy result of shape (F,), which is turned into (F, 1), a transposed layout. Storing it or broadcasting it against (1, F) operands then gives wrong shapes. The rule should apply only when axis covers all non-partition axes.

kwargs.setdefault("keepdims", kwargs.pop("keep_dims", False))
return self._reduce(x, axis, np.sum, "sum", **kwargs)

def max(self, x: NDArray, axis, **kwargs):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sum maps keep_dims to keepdims, but max/min/mean do not. The new reductions are also missing from NKI_NAMESPACES, so the tracer and visualizer never see them.

nl.max(x, axis=1, keep_dims=True): keep_dims falls into **kwargs and lands in NDArray kwargs, so the result is (P, 1) only through the heuristic, and for 3-D inputs keepdims is dropped. Softmax-style kernels using nl.max, nl.add or nl.subtract produce no records for those ops, while nl.sum does.

):
key = np.clip(key, 0, max(ndarray.shape[dim] - 1, 0))
clipped.append(key)
return ndarray[tuple(clipped)].shape

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tile_shape gets the shape by actually fancy-indexing the array, which copies the whole tile. It runs 2 to 3 times per load or store.

Builder.store calls tile_shape and then tile_mask, which calls tile_shape again. The tracer's _nki_offsets_and_mask does it again on the offsets array. That is several extra full-tile gathers on every load/store in the hot per-program loop. Compute the shape from key shapes and slice lengths (np.broadcast_shapes for array keys), and pass the computed shape into tile_mask.

),
Store: lambda dst, keys, value, *, mask=None, **_kwargs: AdapterResult(
Store: lambda dst,
keys=None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whole-tensor access is patched by giving keys=None defaults in three places: Builder.load, Builder.store, and both adapters. Making NKITransformer always emit an explicit keys argument would fix it once.

Because the default sits at the callee, the builder signature (dst, keys, value) no longer lines up with NKI's (dst, value), which causes the positional-store bug above. Keyword dst/src are also not covered. If the transformer normalized every nl.load/store/load_transpose2d call to (tensor, keys, ...), inserting full slices or handling keyword dst/src, the special-case defaults would not be needed.

scope.restore()


def full_keys(tensor):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new full_keys helper repeats the inline full-key construction that load_transpose2d already has, and load_transpose2d was not switched over to it.

load_transpose2d still builds tuple(slice(None) for _ in range(src.data.ndim)) itself, so the same logic now lives in two places that can drift apart. Have load_transpose2d call full_keys(src).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants