Repository navigation
Conversation
…terpreter The legacy NKI interpreter failed on the idioms of AWS's NKI samples: - `nl.load(x[...])` / `nl.store(o[...], ...)` nested in another call was not rewritten to the (tensor, keys) form, so the call lost its keys; - `nl.load(t)` and `nl.store(t, value=v)` without an index had no keys at all; - masks broadcast to the tile in NKI (a [128, 1] row mask on a [128, F] tile), but the interpreter and the tracer required the tile's exact shape, and the tracer failed on a load without a mask; - a kernel that allocates its output in shared HBM and returns it returned None, because the runner dropped the kernel's return value; - reductions took a list of axes, which numpy rejects, and dropped the last free axis, where NKI keeps [P, 1]; - max, min, mean, add, subtract, divide, maximum, minimum, static_cast, par_dim, sequential_range, and static_range were not patched, so they ran NKI's real implementation and failed outside an NKI kernel. Load/store keys default to the whole tensor, masks and stored values are broadcast to the tile shape (`tile_shape` / `tile_mask` in masked_load_store.py), the runner returns the kernel's outputs as numpy arrays, and the missing operations are implemented with NKI's semantics (dtype casts, masked reductions, the [P, 1] reduction shape). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Performance Benchmark
Iterations: 1 warmup + 20 measured |
A client whose pre_run or post_run callback returns False stops the remaining programs; the runner then returned None instead of the outputs the programs so far wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| None stays None.""" | ||
| if mask is None: | ||
| return None | ||
| return np.broadcast_to(np.asarray(mask, dtype=bool), tile_shape(ndarray, keys)) |
There was a problem hiding this comment.
tile_mask/tile_shape only clip integer-array keys, so for an out-of-bounds slice key the 'tile shape' is numpy's truncated slice shape, and broadcasting the full-tile mask to it raises. Masked tail loads and stores that use slices now crash, where before this change they worked.
a.shape == (100, 4), nl.load(a[0:128, :], mask=m) with m.shape == (128, 4): tile_shape returns (100, 4) and np.broadcast_to((128,4) -> (100,4)) raises ValueError (I reproduced this). Before, masked_load's padded slow path returned a (128, 4) tile. The same failure hits Builder.store (value and mask are broadcast to (100, 4)) and the tracer's _nki_offsets_and_mask. Fix: compute the shape from slice.stop as _calculate_target_shape does, not from ndarray[keys].
| ndarray = dst.data | ||
| value_array = value.data | ||
| mask_value = getattr(mask, "data", mask) if mask is not None else None | ||
| keys = full_keys(dst) if keys is None else keys |
There was a problem hiding this comment.
With keys made optional, a whole-tensor store that passes the value positionally (nl.store(out, v), which matches NKI's store(dst, value) signature) binds v to keys, leaving value=None.
nl.store(out_tensor, result_tile): the transformer does not rewrite it (arg0 is not a Subscript), so the call becomes Builder.store(out_tensor, keys=result_tile, value=None). numpy_keys is then a float array, tile_shape indexes with it and raises IndexError. The NKI Store adapter in frontend/nki.py has the same mis-binding, so the tracer crashes first.
| value=new_call, | ||
| type_comment=getattr(node, "type_comment", None), | ||
| ) | ||
| def visit_Call(self, node: ast.Call) -> ast.AST: |
There was a problem hiding this comment.
The rewrite only looks at a positional first argument. nl.store(dst=o[i], value=v) and nl.load(src=x[i]) keep the subscript, so they act on an NDArray copy made by getitem.
nl.store(dst=out[i*128 + ix, iy], value=t): NDArray.getitem returns a copy, and Builder.store then uses full_keys on that copy. The store goes into a temporary and is lost without any error, so the returned output stays uninitialized.
| same_size = value_array.size == int(np.prod(tile_shape_)) | ||
| if value_array.shape != tile_shape_ and same_size: | ||
| # e.g. a [P, 1] reduction stored into a 1-D slice | ||
| value_array = value_array.reshape(tile_shape_) |
There was a problem hiding this comment.
store reshapes any value whose element count matches the tile, whatever its shape. Real shape mismatches are silently reinterpreted instead of raising.
Storing a (4, 8) tile into an (8, 4) slice (for example a forgotten transpose) passes same_size and is reshaped row-major. Scrambled data is written with no error, while NKI would reject the shape. Limit the reshape to a size-1 axis being dropped or added (e.g. [P,1] -> [P]).
| ): | ||
| key = np.clip(key, 0, max(ndarray.shape[dim] - 1, 0)) | ||
| clipped.append(key) | ||
| return ndarray[tuple(clipped)].shape |
There was a problem hiding this comment.
tile_shape does not handle out-of-bounds scalar integer keys. It also lines keys up with array dims by key position, which is wrong once a None (newaxis) key appears.
A masked nl.load(a[r, :], mask=False_mask) with r >= a.shape[0] now raises IndexError in tile_shape, although masked_load's slow path handles int keys via _calculate_target_shape. With keys (None, idx), idx is clipped against ndarray.shape[1] but indexes axis 0. For a.shape == (4, 100) and idx = arange(50), the clip leaves values up to 49 and indexing raises IndexError.
| value = np_func(x.data, axis=_axis(axis), keepdims=keepdims, **reduce_kwargs) | ||
| if not keepdims and np.ndim(value) == 1 and x.data.ndim >= 2: | ||
| # NKI keeps a free dimension: reducing all free axes of a [P, ...] tile gives [P, 1] | ||
| value = value[:, None] |
There was a problem hiding this comment.
The rule that keeps the free dim adds a trailing axis to any 1-D result from a 2-D or larger input. That includes reducing the partition axis.
nl.sum(x, axis=0) on a (P, F) tile gives a numpy result of shape (F,), which is turned into (F, 1), a transposed layout. Storing it or broadcasting it against (1, F) operands then gives wrong shapes. The rule should apply only when axis covers all non-partition axes.
| kwargs.setdefault("keepdims", kwargs.pop("keep_dims", False)) | ||
| return self._reduce(x, axis, np.sum, "sum", **kwargs) | ||
|
|
||
| def max(self, x: NDArray, axis, **kwargs): |
There was a problem hiding this comment.
sum maps keep_dims to keepdims, but max/min/mean do not. The new reductions are also missing from NKI_NAMESPACES, so the tracer and visualizer never see them.
nl.max(x, axis=1, keep_dims=True): keep_dims falls into **kwargs and lands in NDArray kwargs, so the result is (P, 1) only through the heuristic, and for 3-D inputs keepdims is dropped. Softmax-style kernels using nl.max, nl.add or nl.subtract produce no records for those ops, while nl.sum does.
| ): | ||
| key = np.clip(key, 0, max(ndarray.shape[dim] - 1, 0)) | ||
| clipped.append(key) | ||
| return ndarray[tuple(clipped)].shape |
There was a problem hiding this comment.
tile_shape gets the shape by actually fancy-indexing the array, which copies the whole tile. It runs 2 to 3 times per load or store.
Builder.store calls tile_shape and then tile_mask, which calls tile_shape again. The tracer's _nki_offsets_and_mask does it again on the offsets array. That is several extra full-tile gathers on every load/store in the hot per-program loop. Compute the shape from key shapes and slice lengths (np.broadcast_shapes for array keys), and pass the computed shape into tile_mask.
| ), | ||
| Store: lambda dst, keys, value, *, mask=None, **_kwargs: AdapterResult( | ||
| Store: lambda dst, | ||
| keys=None, |
There was a problem hiding this comment.
Whole-tensor access is patched by giving keys=None defaults in three places: Builder.load, Builder.store, and both adapters. Making NKITransformer always emit an explicit keys argument would fix it once.
Because the default sits at the callee, the builder signature (dst, keys, value) no longer lines up with NKI's (dst, value), which causes the positional-store bug above. Keyword dst/src are also not covered. If the transformer normalized every nl.load/store/load_transpose2d call to (tensor, keys, ...), inserting full slices or handling keyword dst/src, the special-case defaults would not be needed.
| scope.restore() | ||
|
|
||
|
|
||
| def full_keys(tensor): |
There was a problem hiding this comment.
The new full_keys helper repeats the inline full-key construction that load_transpose2d already has, and load_transpose2d was not switched over to it.
load_transpose2d still builds tuple(slice(None) for _ in range(src.data.ndim)) itself, so the same logic now lives in two places that can drift apart. Have load_transpose2d call full_keys(src).
Summary
The legacy NKI interpreter (
frontend="nki") runs only kernels written in a restricted style. Kernels in the style of AWS's NKI samples fail before their first tile operation. Of seven tutorial kernels from the samples (softmax, rmsnorm, two layernorms, transpose2d, average pool 2-D, SPMD tensor addition), only the SPMD addition ran under the tracer on main. The failures come from a few idioms:nl.store(o[i], value=nl.copy(nl.load(x[i]))): a load or store nested in another callNKITransformerrewrotenl.load(x[...])tonl.load(x, keys)only when the call was a whole statement, so the nested call lost its keys:TypeError: missing ... 'keys'nl.load(t),nl.store(t, value=v): whole-tensor accessnl.load(x[i * 128 + ix, iy], mask=(i * 128 + ix < n)): a[128, 1]row mask on a[128, F]tilemasked_loadasserts that the mask has the tile's shapenl.load(x[...])without a maskmask.dataofNoneout = nl.ndarray(..., buffer=nl.shared_hbm)...return outNKIInterpretedFunction.rundropped the kernel's return value, so the launch returnedNonenl.max(t, axis=[1], keepdims=True),nl.sum(t, axis=[1])[128, F]tile gives[128, 1], sot / nl.sum(t, axis=[1])broadcasts in NKI and failed herenl.max,nl.min,nl.mean,nl.add,nl.subtract,nl.divide,nl.maximum,nl.minimum,nl.static_cast,nl.par_dim,nl.sequential_range,nl.static_rangeRuntimeError: calling NKI API outside of NKI kernels is not supportedThis PR:
NKITransformer.visit_Callrewritesnl.load/nl.store/nl.load_transpose2dwith a subscripted first argument wherever the call appears.full_keys), in the interpreter and in the frontend'sLoad/Storeadapters, so clients see the same keys.tile_shape/tile_mask(incore/masked_load_store.py) give the shape ofndarray[keys]even when some lanes are out of bounds, and broadcast an NKI mask to it. The interpreter broadcasts masks and stored values with them; the tracer records the broadcast mask, or an all-true mask for an access without one.returndropped them.Builder._reduce: a list of axes becomes a tuple,maskbecomes numpy'swhere(with the dtype's lowest or highest value as the identity of max / min),dtypecasts the result, and a result that would lose its last free axis gets[P, 1], as NKI's simulator gives. Element-wise operations takedtype;nl.loadtakesdtypetoo.par_dimreturns the size, and the three ranges iterate likerange.The shapes and results were checked against NKI's own CPU simulator (
neuronxcc.nki.simulate_kernel). For example,nl.sumover axis[1]of an[8, 4]tile gives(8, 1); over axes[1, 2]of an[8, 4, 3]tile,(8, 1)withoutkeepdimsand(8, 1, 1)with it. With this PR, six of the seven tutorial kernels run under the tracer. The seventh, layernorm v2, usesnisa.bn_stats, which the interpreter does not implement yet.Tests
tests/nki/test_nki_kernel_idioms.py: four kernels in the samples' style run under TileLens and undernki.simulate_kernel; their outputs must match. The kernels cover a masked softmax over 200 rows, whole-tensor and nested loads and stores, an SPMD addition on a 2×2 grid, and a kernel with loops, casts, and binary operations. All four fail on main. A fifth test stops the grid after the first program with a client whosepost_run_callbackreturns False and checks that the launch still returns that program's output; it fails without the second commit.tests/nki/test_nki_tracer.py::test_tracer_records_reduce_sumnow expects the reduction's output shape(4, 1)instead of(4,), as NKI's simulator gives. Storing that result into the test's 1-D output slice still works: a stored value with the tile's number of elements is reshaped to the tile.pytest tests/ -m ""(NKI tests included): 621 passed, 2 skipped on main; 626 passed, 2 skipped with this PR. Environment: Python 3.12, Triton 3.8.0, neuronx-cc 2.22.12471, nki 0.2.0.ruff checkandruff format --check(v0.1.9) pass on the changed files.🤖 Generated with Claude Code