Conversation
…r the 13 LAPACK factorisations
Adds the FIRST committed differential-fuzz (bit-exact-vs-NumPy) corpus coverage for the
LAPACK factorisation family. Until now these were gated only by unit tests (API/error
contracts) + the live-numpy interop suite; the CLAUDE.md note that they were "deliberately
absent from the differential-fuzz corpus" is now resolved — they have a committed, replayed,
byte-exact gate like every other op.
Ops covered (264 cases): cholesky, eig, eigvals, eigh, eigvalsh, svd, svdvals, pinv,
matrix_rank, cond{None,2,-2}, lstsq, qr{reduced,complete,r,raw} and norm{2,-2,'nuc'} —
across dtypes (float64/complex128/float32 + int/bool widen-to-float64), memory layouts
(C/F/negrow/negcol/stride2/slice), shapes (tall/wide/square/1x1/empty), batched stacks, and
the parameter surface (upper, UPLO L/U, full_matrices, compute_uv, rcond, tol/rtol, ord, axis,
keepdims). Tuple results (svd/eig/eigh/qr/lstsq) ride the kind:"tuple" comparator (arity
asserted); array siblings ride the ordinary bytes contract.
HOST-PINNED exactly like the matmul_parity tier, and for the same reason: NumSharp.Core ships
NO managed LU/QR/SVD/eigensolver, so these compute ONLY through the opt-in
NumSharp.Interop.OpenBLAS backend, and the result bytes come out of a specific LAPACK build
dispatched to a specific CPU kernel. The gate enables that backend before replay and Disable()s
after; a host that cannot load NumPy 2.4.2's pinned scipy-openblas (matched by CONTENT sha256,
not file name) goes Inconclusive, never red. Pinned at threads=1 — the deterministic config the
interop live-parity suite proves — so gen_linalg_parity() forces single-thread via ctypes and
the recorded bytes are threading-independent. NumPy's linalg is "lite" (factorises every operand
in double/cdouble and rounds back once via _commonType), so float32 results are byte-identical
too. Empirically probed 25/26 byte-exact on this host before wiring; the sign/phase freedom of
eigenvectors/U/Vh/Q/R is resolved identically on both sides because the SAME LAPACK routine runs.
Result: 264/264 bit-exact on the pinned host; the gate was teeth-verified (a one-byte corpus
corruption turns it red with a precise diagnostic).
Only the byte-REPRODUCIBLE surface is recorded. Three factorisation outputs are NOT, and are
deliberately excluded (covered by the interop suite's reconstruction/tolerance checks instead):
* complex-Hermitian eigh EIGENVECTORS — heevd does not canonicalize the phase and it is not
reproducible across processes; complex-Hermitian eigenVALUES are recorded via eigvalsh.
* float32 eig/eigvals with COMPLEX eigenvalues — NumPy yields complex64, NumSharp complex128
(no complex64 dtype); float32 eig is recorded only for all-REAL-eigenvalue matrices.
* cond/norm orders that are NOT SVD-based (fro/1/-1/±inf) — they compose an elementwise
reduction whose summation order rounds 1 ULP off NumPy (measured: cond(a,'fro') differs in
the last byte); only the SVD-based orders (cond None/2/-2, norm 2/-2/'nuc') are recorded.
A stale claim was also corrected while probing: complex pinv is now byte-exact (the
SvdLstsqLiveParityTests comment saying otherwise predates complex128 products being routed
through zgemm — the reconstruction matmul is byte-exact now).
Implementation:
* test/oracle/gen_oracle.py — gen_linalg_parity() + _set_openblas_threads()/_lp_* helpers
(reuses describe/_arr_expected/_tuple_expected/_mp_layout); mode=="linalg_parity" in main()
writes linalg_parity.jsonl + linalg_parity.host.jsonl (blas_identity records threads=1).
* OpRegistry.cs — array cases (cholesky/eigvals/eigvalsh/svdvals/pinv/matrix_rank/cond/norm,
svd compute_uv=false, qr mode='r') + ParseUplo/ParseOrd helpers.
* OpRegistry.Kinds.cs — tuple cases (svd/eig/eigh/qr/lstsq) in ApplyTuple.
* MatmulParityPin.cs — Load(fileName) generalized + Tier field so the Inconclusive diagnostics
name the right regeneration command; guards both host-pinned tiers.
* FuzzCorpusTests.cs — LinalgParity() [DoNotParallelize] test + MinCases floor (210).
* Fuzz/README.md + .claude/CLAUDE.md — documented the new tier and its exclusions.
…_path, cov/corrcoef, and the polynomial family (17 fns)
Adds committed bit-exact-vs-NumPy corpus coverage for the four function groups that had
none, splitting them by their real byte-exactness (empirically probed, NumSharp-vs-NumPy):
portable ops go in portable tiers; the three backend-required polynomial ops join the
host-pinned linalg_parity tier.
NEW portable tiers:
* poly.jsonl (gen_oracle.py poly) — the PORTABLE polynomial family: poly (1-D roots ->
coefficients), polyval (Horner), vander, polyder, polyint, polyadd/polysub/polymul,
polydiv (quotient+remainder tuple), poly1d (leading-zero normalisation + construction
from roots). Pure array arithmetic / convolution / Horner — NO backend, NO long
reduction — so bit-exact everywhere (probed: Horner order, leading-zero normalisation,
polynomial long division all match byte-for-byte) across float64/float32/complex128 +
small-exact int64 + strided/reversed reads. poly/polyint on int64 return float64 (NumPy
floats them); polyder/vander preserve int64 — the corpus records NumPy's actual dtype.
* einsum.jsonl — np.einsum (integer/complex-integer contractions + small-exact float
contractions + the whole view path: transpose/diagonal/trace/no-sum/copy) and
np.einsum_path (the planner's info STRING, text kind, shape-derived, non-ellipsis,
byte-identical to NumPy). einsum operands are kept NONZERO on purpose: a signed zero
diverges in outer/hadamard (NumPy's sop accumulator, seeded +0.0, absorbs a -x*0=-0.0
term into +0.0 while NumSharp's element-wise multiply keeps the raw -0.0); larger float
contractions route through matmul (NumPy's default einsum uses its own C iterator) and
are NOT byte-exact, so operands stay small-exact.
EXTENDED tiers:
* products.jsonl gained cross (the lone product-family gap — multiply-subtract, no
reduction, bit-exact f64/f32/c128/i64 at every value/layout; int32-and-narrower widen to
int64 in NumPy 2.x cross, a dtype divergence left out) and cov/corrcoef (normalized dot,
byte-exact for the SMALL observation counts here across rowvar/bias/ddof/y/complex/
int-widen; the WEIGHTED fweights/aweights path rounds 1 ULP off in the fact
normalisation and is left to cov's tolerance battle-tests).
* linalg_parity.jsonl (host-pinned) gained roots (eigvals of the companion matrix),
polyfit (lstsq) and poly of a 2-D matrix (char poly via eigvals) — all THROW without the
OpenBLAS backend, so they cannot be portable. Small operands, threads=1, byte-exact.
MisalignedRegistry: the pure single-operand arithmetic ops (poly/polyder/polyint/vander/
poly1d_coeffs/poly1d_fromroots/cov/corrcoef) are CARVED OUT of the blanket "unary ~ULP"
excuse — they are arithmetic, not transcendental-libm, so the excuse's rationale does not
apply and a <=2-ULP drift is a regression that must fail the gate (the same narrowing the
ported float32 kernels get). Verified: the clean tiers pass byte-exact, and a 1-ULP
corruption of a poly case now turns the gate red (teeth-checked); without the carve-out it
was silently excused.
Implementation:
* test/oracle/gen_oracle.py — gen_poly(), gen_einsum(), the cross/cov/corrcoef block in
gen_products, and roots/polyfit/poly(2-D) in gen_linalg_parity (+ _lp_poly_fit helper);
modes "poly"/"einsum" wired into main().
* OpRegistry.cs — Apply cases (cross/cov/corrcoef/einsum/poly/roots/polyfit/polyval/vander/
poly1d_coeffs/poly1d_fromroots/polyder/polyint/polyadd/polysub/polymul) + ParseUplo/
ParseOrd already present. OpRegistry.Kinds.cs — polydiv (tuple), einsum_path (text).
* FuzzCorpusTests.cs — Poly()/Einsum() tests + MinCases floors (poly 60, einsum 35,
products 310, linalg_parity 220).
* Fuzz/README.md + .claude/CLAUDE.md — documented the tiers, the carve-out and every
divergence.
Gates green: full FuzzCorpusTests 57/57 (no regression); the four affected tiers byte-exact;
generators deterministic (identical sha256 on re-run); products/linalg_parity diffs are
append-only.
…tion family — solve/inv/det/slogdet/tensorinv/tensorsolve + matrix_power(n<0) The seven OpenBLAS-dependent np.linalg members reachable through getrf/gesv had ZERO committed differential-fuzz coverage: solve, inv, det, slogdet, tensorinv, tensorsolve and matrix_power(n<0). They were implemented and verified once (135/135 by a one-off script) but never pinned in a corpus, so a build/kernel/thread drift or a code regression would have gone unnoticed. The FFT family — all 18 np.fft.* — was AUDITED and found already complete (fft.jsonl, 2000 cases, portable pocketfft), so the LU family was the remaining 'other OpenBLAS dependent functions' gap. Adds +90 cases to the host-pinned linalg_parity tier (276 -> 366). Unlike the eigen/SVD factorisations, LU with partial pivoting is a deterministic function of the input — NO sign/phase ambiguity — so EVERY output is byte-reproducible (probed 2/2 per case, cross- process) and nothing is excluded. det of one matrix is a 0-D scalar and of a stack is 1-D; slogdet is a kind:tuple (sign, logabsdet) whose complex sign is a unit-modulus complex and whose logabsdet stays real; a singular operand gives det->0 and slogdet->(0,-inf) exactly. solve honours NumPy 2.0's b-is-a-vector-iff-1-D rule (vector / matrix / batched-matrix / broadcast-vector RHS); tensorinv/tensorsolve are reshape->inv/solve->reshape; matrix_power with negative n is inv(a)**|n| and is host-pinned (portable positive/zero n stays in products.jsonl). float32 upcasts to double and rounds back (_commonType) so it is byte-exact too; int/bool widen to float64. Verified across dtypes x layouts x shapes x batched x degenerate. Wiring: gen_oracle.py gains the LU-factorisation block in gen_linalg_parity plus a new _lp_arr2 (two-operand array-result helper, for solve/tensorsolve — a in layout, b C-contig); OpRegistry.cs::Apply gains inv/det/solve/tensorinv/tensorsolve (the matrix_power case already handled any n); OpRegistry.Kinds.cs::ApplyTuple gains slogdet; the linalg_parity MinCases floor moves 220 -> 340 to lock the new coverage. README.md ledger + .claude/CLAUDE.md updated. 366/366 byte-exact on the pinned host (Win/AMD64/Haswell scipy-openblas sha 74a4...); host pin unchanged (same machine); LinalgParity + Products + MatmulParity + HarnessSelfTests all green, Skipped=0.
Adds the NumPy `ndarray.itemsize` attribute — "Length of one array element in bytes" — to NDArray, the last missing sibling of the size/ndim/nbytes/dtypesize metadata family. What it is ---------- A pure O(1) property of the dtype: `public int itemsize => Storage.DTypeSize`. NumPy surfaces `PyArray_ITEMSIZE` (the dtype's element size) as a plain Python int on every ndarray; it is independent of shape, strides, offset and layout, so every view of a given dtype reports the same value. No kernel / NDIter / ILKernelGenerator is involved (there is no loop over element data) — it mirrors the existing `dtypesize`/`nbytes` properties, reading the same `Storage.DTypeSize` jump-table that already backs them. Why an alias rather than a rename --------------------------------- NumSharp has long carried `dtypesize` (legacy name) for the identical value. `itemsize` is added ALONGSIDE it (existing API untouched) so Python code ports verbatim (`arr.itemsize`) while every current `dtypesize` call-site keeps working. `itemsize == dtypesize`, and `size * itemsize == nbytes`, both asserted. Parity (probed against NumPy 2.4.2) ----------------------------------- Byte-identical to NumPy for the 13 dtypes with a NumPy analog: bool/int8/uint8→1, int16/uint16/float16→2, int32/uint32/float32→4, int64/uint64/float64→8, complex128→16. The two NumSharp-only dtypes with no NumPy analog report their in-memory element size (Char→2, Decimal→16), consistent with `dtypesize`/`nbytes`. Verified layout-invariant across C-contiguous, 0-d, empty, strided, transposed, negative-stride, broadcast and post-astype views — exactly as NumPy behaves. Tests ----- NDArray.flags.Test.cs gains an itemsize section (3 methods, 24 total in the class, all green): all-15-dtype coverage, layout invariance, and the alias/nbytes-composition identities. Gated by unit tests like its `dtypesize`/`nbytes` siblings (a scalar dtype attribute produces no (dtype,shape,bytes) result for the NDIter differential-fuzz corpus).
…+ layout matrix ndarray.nbytes (Backends/NDArray.cs:501, committed in 792a9f1) is a faithful 1-to-1 port of NumPy's PyArray_NBYTES = PyArray_ITEMSIZE * PyArray_SIZE, i.e. `size * itemsize`. Re-verified flawless against NumPy 2.4.2 across the full variation matrix and captured that verification as committed regression tests so the "flawless" guarantee is durable (the implementation itself is untouched — already minimal and correct). Empirical parity (probed against numpy==2.4.2 this session) ----------------------------------------------------------- - 35/35 bit-exact over 13 NumPy dtypes x 22 layouts: C/F-contiguous, strided, transposed, negative-stride, simple slice, sliced+composed, broadcast, partial broadcast, scalar 0-d, 0-d view, one-element 1-D, empty, empty+composed, newaxis, singleton dim, rank-5, fancy-indexed, boolean-mask, reshaped view, astype(int16), astype(complex128). - Char/Decimal (the two NumSharp-only dtypes, no NumPy analog) follow the same formula: 30 elems -> 60 / 480 bytes (itemsize 2 / 16). - size*itemsize==nbytes identity holds across all 15 dtypes on a strided+ negative-stride view. - 64-bit correctness (no int32 truncation): a broadcast (1,1)->(100000,100000) int64 view reports 80000000000, byte-identical to NumPy's npy_intp result. Both sides compute in signed 64-bit, so even the (unrealizable) overflow regime agrees. Tests added (Backends/NDArray.flags.Test.cs, +2 methods -> class now 26 green) ------------------------------------------------------------------------------ - Nbytes_AllDtypes_MatchNumpy: all 15 dtypes on (3,5,2)=30 elements, values probed from NumPy 2.4.2 (13) plus the two NumSharp-only dtypes. Fills the direct dtype-coverage gap (previously only Complex was tested directly; the all-dtype guarantee rode transitively on the itemsize test + a float64 size*itemsize==nbytes identity). - Nbytes_LayoutMatrix_MatchNumpy: the layouts the prior 4 nbytes tests did not reach directly — F-contiguous, transposed, negative-stride, partial broadcast, rank-5 (all logical-size 96), plus copying selections (fancy-index/boolean-mask -> 48, one-element 1-D / 0-d view -> 4) and astype (int16 -> 48, complex128 -> 384). All values probed against NumPy 2.4.2. No production code changed; the concurrent ndarray.strides WIP in NDArray.cs is left untouched.
…moryView) Implements NumPy's `ndarray.data` (probed against NumPy 2.4.2), the read-only attribute that is literally `memoryview(self)` — "Python buffer object pointing to the start of the array's data." Closes the coverage gap where `.data` was only "partial" (loosely mapped to the typed `Data<T>()` method, which is NOT a buffer object). WHAT - New `NDArray.data` (lowercase, matching the whole np.* surface: shape/size/flat/ device/strides). Read-only property, so it cannot be assigned — mirroring NumPy, whose `.data` is getter-only (assignment raises AttributeError). A fresh handle is returned per access, exactly as NumPy hands out a new memoryview each time. - New house type `np.MemoryView` (sealed, nested under `np` like np.Broadcast / np.FlatIterator / np.NDIterator) — the NumSharp analog of Python's `memoryview`. It OWNS NO MEMORY: every member reads LIVE through the source array's Storage/Shape (per the "back it with the existing buffer infra" design decision), and holds the source array as `obj` to keep it alive (NumPy's `memoryview.obj`). SURFACE (all bit-identical to NumPy 2.4.2 — see gate): obj, itemsize, ndim, nbytes (logical size·itemsize), @readonly (true for broadcast views), shape, strides (in BYTES — NumSharp strides are elements, ·itemsize here), format (struct code), c_contiguous / f_contiguous / contiguous, Length (len(mv)), Pointer (void*) / Address (IntPtr) at the LOGICAL first element, write-through scalar element get/set via this[params long[]], tobytes(order=C/F/A), hex(). LOAD-BEARING DETAILS (probed, easy to get wrong): - Pointer = Storage.Address + Shape.Offset·itemsize — the logical start, matching NumPy's a.data / PyArray_DATA / a.ctypes.data / __array_interface__['data'][0]. Verified across offset ([3:7] -> base+12) and reversed ([::-1] -> base+36, the last physical element) views. The offset lives in Storage.Address OR Shape.Offset (mutually exclusive across NumSharp's view paths), so the single formula is robust. - strides in BYTES via Shape.Strides (elements) · itemsize — NOT NDArray.strides — so a stride-0 broadcast axis stays 0 and a reversed view stays negative. - format follows numpy/_core/src/multiarray/buffer.c EXACTLY, incl. the platform split: NPY_LONG emits 'l'/'L' on Windows LLP64 but 'i'/'I' elsewhere (numpy's int32 is NPY_LONG on Windows, NPY_INT on LP64), so int32/uint32 are platform-dependent. Char (2-byte UTF-16) and Decimal (opaque 16B) have no NumPy dtype -> documented house codes 'u' / '16s'. - tobytes gathers in LOGICAL order via the existing copy machinery (NDIter/IL copy kernels), with a direct-memcpy fast path when already contiguous in the target order; 'A' = physical order if contiguous (F when F-contig, else C). - Write-through element set coerces the value to the dtype (a 1-element astype, the standard NumSharp scalar-cast path — Storage.SetValue demands an exact CLR type), and refuses a readonly (broadcast) target. Documented divergence: astype WRAPS an out-of-range integer where NumPy's memoryview raises; use arr.flatiter for the weak-scalar bounds check. NOT MODELLED (documented boundaries, not gaps): partial-index sub-memoryviews, cast, tolist, release / the context-manager protocol, element iteration — NumSharp has no Python buffer protocol, so these Python-object conveniences have no counterpart; the buffer ESSENCE (pointer + metadata + write-through + tobytes) is what data is for. GATE: test/NumSharp.Tests/APIs/np.memoryview.Test.cs (17 tests, net8.0 + net10.0), probed against NumPy 2.4.2. Differentially validated bit-exact vs numpy across {1d,2d,offset,reversed,strided,transposed,broadcast,0-d,empty,f64} × {ndim,itemsize, nbytes,readonly,shape,byte-strides,format,contiguity,tobytes C/F/A} and all 15 dtype format codes. As a pure O(1) accessor (a buffer handle, not a value-producing NDIter op) it is unit-test gated — same rationale as flatiter/nditer/ndindex/device — and deliberately absent from the differential-fuzz corpus. Perf: handle is O(1) (~14 ns); tobytes contiguous fast path is memcpy-bound at ~1.26x NumPy.
…m guard
Validated the whole ndarray.* instance surface (T/base/device/dtype/dot/cumsum/copy/
arg*/astype/shape/size/strides/ndim/mT/sort/ravel/item/max/mean/min/flatten/std/sum/
swapaxes/reshape/resize/partition/prod/to_device/tobytes/tofile/tolist/transpose/var/view)
against NumPy 2.4.2. 38/40 were already bit-aligned (values, NEP50 int64 sums, view/copy
semantics, error paths). Two fixes:
1) ndarray.strides now returns BYTES per axis (element strides x itemsize), matching NumPy's
PyArray_STRIDES. Previously returned element strides. The public property lives in
Backends/NDArray.cs; the internal Shape.Strides/Shape.strides field stays in ELEMENTS
(ARCHITECTURE INV-02 unchanged). 8 kernel files that read the property expecting elements
are migrated X.strides -> X.Shape.Strides (value-identical, zero-alloc): Default.Dot.NDMD,
Default.ClipNDArray (incl. loCast?/hiCast?), Default.Reciprocal, Default.Reduction.{CumAdd
(also input./ret.),CumMul,Std,Var}, np.array (@out jagged-2D path x4). An exhaustive
src-wide .strides audit confirms every other receiver is Shape-typed (unchanged).
BREAKING: external code reading ndarray.strides now gets bytes (NumPy behaviour).
2) item(long i,long j) and item(long i,long j,long k) now validate ndim. They previously
called Storage.GetValue with no rank check, so e.g. a.item(1,2) on a 3-D array silently
returned a wrong element; NumPy raises ValueError 'incorrect number of indices for array'.
Tests updated to byte semantics (all asserted the OLD element strides):
NDIterNumPyParityTests 5x NegativeStride_* (int64 x8), NDIterZeroDimOpAxesTests ByteStride
helper (was strides*itemsize -> double-counted; now Shape.Strides*itemsize), OpenBugs
Bug_SwapAxes_Strides_* (x3, int64) + Bug_SliceBroadcast_CopyWorkaround (int32 [3,1]->[12,4]) --
these were PASSING guards whose 'Bug_' names/comments were stale (swapaxes/copy already produce
correct strides). ShapeAssertions.HaveStrides and np.broadcast ret[N].strides are Shape-based
(element) and correctly untouched.
Verification: FuzzMatrix differential gate 78/78 bit-exact vs NumPy; full CI-style suite
(net10.0, excl OpenBugs/HighMemory) 13577 passed / 2 pre-existing baseline failures
(T1_33_AsNumpyDtypeName_Char_MisreportsSize, NoOperationDefersItsBufferRelease) / 11 skipped.
… dtype= (NumPy 2.4.2 parity)
Audited the ndarray instance-method family (all/any/clip/compress/conj/conjugate/
cumprod/cumsum/diagonal/nonzero/put/repeat/round/searchsorted/squeeze/take/trace)
against NumPy 2.4.2. Found three gaps; the other 13 methods matched exactly and are
unchanged (signatures, view/read-only semantics, dtype promotion, error paths all
re-verified via probes).
1. searchsorted — CORRECTNESS BUG (all three overloads: int, double, NDArray).
NumSharp cast the key `v` DOWN to the sorted array's dtype before searching
(Converts.ChangeType(v, a.typecode)), truncating/wrapping it:
[1,2,3,4].searchsorted(2.5) -> 1 (NumPy 2; 2.5 was floored to 2)
[1,2,3,4].searchsorted(4.5) -> 3 (NumPy 4)
int8[1,2,3,4].searchsorted(300) -> wrong (NumPy 4; 300 wrapped into int8)
uint8[1,2,3,4].searchsorted(-5) -> wrong (NumPy 0; -5 wrapped into uint8)
NumPy's PyArray_SearchSorted promotes BOTH a and v to result_type(a,v) and
searches there. Now: common = promote_types(a.typecode, v.typecode); cast a (and
v) to it; search. The same-dtype path is byte-identical (promote(T,T)=T), so the
sort differential-fuzz tier (401 cases) and 131 searchsorted unit tests stay green.
np.digitize already carried a hand workaround for this exact bug (pre-promoting
bins via _FindCommonType) — now redundant but left intact and harmless.
2. cumsum / cumprod — SIGNATURE PARITY: add the out= parameter that NumPy's
a.cumsum(axis, dtype, out) / np.cumsum(a, axis, dtype, out) carry, on BOTH the
free function and the instance method. out receives the result via NumPy's UNSAFE
casting (float->int truncates, complex->real drops the imaginary part) and is
returned; a wrong-shape out raises IncorrectShapeException. Shared WriteScanToOut
helper. Closes the "cumprod omits out=" deviation noted when the wrappers landed.
3. clip — KWARG PARITY: expose dtype= on the instance method (NumPy's
ndarray.clip(min, max, out, **kwargs) forwards dtype; the np.clip free function
already supported it). Documented that clip() with neither bound returns a copy in
NumPy 2.x (the old "One of max or min must be given" ValueError is gone).
Test cleanup: Bug12_Searchsorted_ArrayInput_WrongResults documented an
already-fixed bug and read the int64 result with GetInt32 (fired a debug assert);
switched to GetInt64 and dropped the now-stale [OpenBugs] marker.
Verified in an isolated worktree at the committed HEAD (no parallel-session WIP):
Core builds clean, 456 instance-method/searchsorted/cumsum/cumprod/clip/digitize
unit tests pass CI-style, Sort + Scan differential-fuzz tiers green.
Gate: test/NumSharp.Tests/Backends/NDArrayMethodParityTests.cs (+12 tests).
…mer, zero-copy Adds np.frombuffer overloads that accept np.MemoryView (from ndarray.data) — the CONSUMER side of the buffer object added in 25ae705, so `np.frombuffer(arr.data)` round-trips: it reconstructs a 1-D array that SHARES arr's memory (probed against NumPy 2.4.2). Completes "memoryview related apis to use memoryview". WHAT - frombuffer(MemoryView, Type dtype=null, long count=-1, long offset=0) - frombuffer(MemoryView, NPTypeCode dtype, long count=-1, long offset=0) [core] - frombuffer(MemoryView, string dtype, long count=-1, long offset=0) mirroring the existing byte[]/Span/Memory overload shapes. SEMANTICS (all bit-exact vs NumPy 2.4.2): - ZERO-COPY: the result views the source's unmanaged buffer; writes through to the source. Reinterprets bytes as `dtype` (int32->float32 == 1.0, int32->int16 doubles the count, int32->int64 halves it — little-endian, byte-identical to NumPy). - Requires a C-contiguous buffer; a strided/transposed/F-contiguous memoryview raises NumPy's "memoryview: underlying buffer is not C-contiguous" (BufferError text; as InvalidOperationException, NumSharp has no BufferError type). - offset is in BYTES, count in ELEMENTS; default dtype float64; 2-D C-contig source flattens to 1-D; 0-d source -> one element; empty -> empty. offset/count validation and messages are IDENTICAL to the existing byte[] path ("buffer size must be a multiple of element size" / "buffer is smaller than requested size"). - Read-only buffer -> read-only array (writeability propagated). IMPLEMENTATION — a pure composition of existing zero-copy VIEW primitives, no data copy, no per-element loop (frombuffer is a reinterpretation), no manual pointer/ARC, no per-dtype compute switch: src.reshape(size) C-contiguous -> distinct 1-D view -> view(dtype) (AliasAs) reinterpret bytes to the target dtype in ELEMENT space -> [offsetElems:+count] slice only when a sub-range is requested The aligned-offset path (offset==0 and every valid offset when the source byte count is a multiple of the target itemsize — i.e. essentially all real usage) stays in element space; a genuinely non-itemsize-aligned offset (only reachable when the source byte count is NOT a multiple of the target itemsize, e.g. a byte source reinterpreted as int16 at an odd offset) falls back to a flat byte-window slice. ARC on the shared MemoryBlock keeps the source buffer alive for the view's lifetime (NumPy's memoryview .base) — verified: the view survives the source being disposed + GC'd. GATE: test/NumSharp.Tests/Creation/np.frombuffer.MemoryView.Test.cs (14 tests, net8.0 + net10.0), probed against NumPy 2.4.2: round-trip/write-through, default float64, reinterpret widths, 2-D flatten, count+offset, 0-d, non-contiguous/misalign/too-big/ null errors, readonly propagation, survives-source-dispose (ARC), all 15 dtypes, Type + string overloads. Existing 25 frombuffer/memoryview tests stay green (no overload ambiguity — MemoryView has no implicit conversions). PERF: zero-copy O(1) (constant regardless of array size — no data copy). Same-dtype ~450 ns/call (one reshape), cross-dtype ~1.6 us/call (the view(dtype)/AliasAs cost). Allocations minimized from 4 view-ops to 1-2. Sub-1.5x vs NumPy's single C view-constructor (~355 ns) because one NDArray/Storage/Shape construction already ~= NumPy's whole call — the documented per-construction floor (cf. fill_diagonal 0.62x, meshgrid-small 0.5x), not a copy/algorithm cost.
…n the oracle tier The searchsorted IL kernel compared Complex by its REAL part only, diverging from NumPy — whose complex searchsorted uses CDOUBLE_LT, the SAME NaN-aware lexicographic (real, then imaginary) total order that np.sort / np.sort_complex already apply. A properly complex-sorted array was therefore not correctly searchable: np.searchsorted([2+1j,2+3j,2+5j], 2+4j) -> NumPy 2/2, NumSharp (was) 0/3 np.searchsorted([2+1j,2+3j,2+5j], 2+3j) -> NumPy 1/2, NumSharp (was) 0/3 key with a NaN imaginary component -> NumPy len(a), NumSharp (was) mid Fix (DirectILKernelGenerator.Search.cs): the kernel no longer remaps Complex to its real double — it loads the full 16-byte struct and compares via NumpyComplexLess, a verbatim CDOUBLE_LT port (the same comparator as AxisSort.ComplexCmp), so searchsorted stays consistent with sort. The monotonic key-bound carry now uses that same NaN-aware comparator, matching binsearch.cpp. Verified 800/800 bit-exact vs NumPy 2.4.2 over random complex arrays (duplicate reals, differing imags, NaN/inf/-0 components) and by the FuzzMatrix gate. isin's complex sort-probe path inherits the fix (Groupa green). Oracle: gen_searchsorted expanded 28 -> 130 cases (sort.jsonl 848 -> 950) across families the tier did not previously cover at all — * duplicates in a: first-vs-last / left-vs-right, plus out-of-range clamp to 0/len(a) * MIXED-DTYPE key promotion (the cc676ea result_type path): fractional / out-of-range / negative keys that a naive down-cast to a's dtype would wrap or truncate (e.g. int8 a + key 300 -> index 8, not a wrapped 44) * sorter: unsorted a + argsort(a), exercising the separate argbinsearch IL kernel * NaN keys and a NaN in a's sorted tail (carried-bound reset) * complex lexicographic (equal real / differing imag / NaN component) * strided / offset / negative-stride a (the contiguousA=false, arrStride!=elemSize path) * empty a (all zeros), scalar v (0-d result), empty v (empty result) The cross-cutting families are gated on len(dtypes) > 1 so char_tier's single-dtype uint16->char weave only picks up the same-dtype base/dup families (verified: 4 clean uint16 cases). OpRegistry.searchsorted now replays an optional sorter as ops[2]; sort.jsonl floor 401 -> 940. Tests: AuditV2 T1_27c de-staled — searchsorted has had the `side` parameter for a while, so the placeholder that asserted "side='right' cannot be expressed" is now a passing left(1)/right(3) regression test (dropped [OpenBugs]). Added explicit complex lexicographic + NaN-component unit tests; corrected the stale "compared by real part" comment. FuzzMatrix green; searchsorted / digitize / isin / sort_complex unit tests green.
…Py 2.x NumSharp's UnmanagedStorage.AliasAs<T> (the storage half of ndarray.view(dtype)) rejected ANY non-C-contiguous array when reinterpreting to a different-size dtype, and rebuilt a fresh C-contiguous shape from the buffer base — discarding the view's offset and outer-axis strides. NumPy 2.x (numpy/_core/src/multiarray/getset.c array_descr_set, since gh-1.23) only requires the LAST axis to be contiguous: every outer axis keeps its byte stride while the last axis is rescaled by the size ratio. So np.arange(12).reshape(3,4)[::2].view(np.int32) — last axis contiguous, whole array not C-contiguous — SUCCEEDS in NumPy (shape (2,8), strides (64,4), shares memory) but threw InvalidOperationException in NumSharp. An existing test (View_NonContiguous_ThrowsForDifferentSize) pinned that non-NumPy behavior. Rewrite AliasAs<T> to NumPy's algorithm: - same itemsize: pure reinterpret keeping dims/strides/offset for ANY layout (contiguous/strided/transposed/negative-stride/broadcast/0-d); the wrap now spans the whole backing slice so an element-dropping strided view addresses every in-bounds coordinate (a latent under-span the old newCount=_shape.size had). - different itemsize: require only the LAST axis contiguous; preserve outer byte strides (as new-dtype element strides) and the base offset; resize the last axis; keep a read-only (broadcast) source read-only. Verbatim NumPy error messages for 0-d, non-contiguous last axis, and non-divisible last-axis byte size. A rare strided+offset view whose byte offset/stride is not a whole multiple of the new itemsize is refused (NumSharp stores strides/offset in elements, not bytes) — still strictly more permissive than before. AliasAs is reached only through view(). Differential-verified bit-exact vs NumPy 2.4.2 across outer-strided (i8->i4, i4->i2/i8), 3-D sliced, i2->i4 widening, and broadcast (read-only) cases incl. values/shape/strides/writeability/write-through; all four error cases match. Char->ushort same-size reduction path (argmax/argmin/max/min, contiguous+strided) verified unaffected. Full suite: 13632 passed, only the 2 pre-existing baseline failures remain (0 regressions). Update ViewTests: replace the wrong-behavior test with View_OuterStridedDifferentSize_Works, View_LastAxisNonContiguousDifferentSize_Throws, View_BroadcastOuterDifferentSize_StaysReadOnly, View_0dDifferentSize_Throws — all pinned to NumPy 2.4.2 output.
…rided inner-contig fast path
Re-audited ndarray.fill against NumPy 2.4.2 across all 13 numpy dtypes x the
coercion-adversarial value set (probed a.fill(v) and diffed C# vs NumPy). The
value/layout matrix was already bit-exact, but the WEAK-scalar (C# primitive)
error taxonomy diverged in four cases -- the coercion mirrors NumPy's
PyArray_FillWithScalar -> PyArray_Pack, which funnels a python scalar through
int()/float():
- NaN -> integer dtype: NumPy raises ValueError 'cannot convert float NaN to
integer' (dtype-independent, same for every int width + f2/f4/f8 source), NOT
an OverflowError. Was OverflowException. Fixed.
- +/-inf -> integer dtype: NumPy raises OverflowError 'cannot convert float
infinity to integer' ('infinity' for BOTH signs). Message was wrong. Fixed
(still System.OverflowException == NumPy's OverflowError).
- complex (weak) -> non-bool integer dtype: NumPy raises TypeError 'int()
argument ... not complex' even for a zero imaginary part. NumSharp was
SILENTLY dropping the imaginary part and storing the real part (wrong result,
no error). Fixed via a case Complex in CheckWeakScalarFitsInteger.
- complex (weak) -> float family (Half/Single/Double/Decimal): NumPy raises
TypeError 'float() argument ... not complex'. Was InvalidCastException
(Complex has no IConvertible). Fixed.
complex -> bool stays truthiness (nonzero -> True), matching NumPy. The STRONG
0-d NDArray path was already correct: NumPy's 0-d-ARRAY fill takes
npy_cast_raw_scalar_item (raw modular cast -- wraps ints, drops complex imag),
which is exactly what NumSharp's nd.astype(dtype) does; only numpy SCALAR
objects (which NumSharp has no analog for) take the value-range-checked setitem.
Also added a strided fast path: a non-contiguous view whose innermost axis is a
unit-stride contiguous run (row slice m[::2], row-strided/offset row, 3D
[:, ::2, :], negative-OUTER-stride [::-1]) is a stack of contiguous runs, now
splatted per-run with the same SIMD UnmanagedSpan<T>.Fill via an outer-axis
odometer (handles negative outer strides), keyed per-dtype through the
reflection-cache and non-allocating (struct span Slice). Works at ANY rank --
NumSharp has no ndim cap, so the odometer stackallocs its coord array for the
common shallow ranks and heap-allocates only past a 64-deep stack budget (the
FiniteScan.cs pattern). Falls back to NDIter.Copy for transposed /
non-unit-or-negative inner stride / broadcast. Byte-identical to NumPy across 12
differential layouts + validated at 64/70/200 dims.
Perf (NPY/NS, warm, Release): contiguous 100K-10M 0.97-1.09x (parity -- the
memory-bandwidth ceiling; 1.5x is unreachable for a pure splat since NumPy
already hits memset speed); small-inner strided 3D 0.89x; large strided [::2]
0.50x (up from 0.41x; residual is NumPy's non-temporal movnt stores, the same
limit as contiguous-10M -- a core-wide Fill concern, not a fill bug).
Tests: NDArray.fill.Test.cs 24 -> 34 (updated NaN case to ValueError; added
inf/complex-int/complex-float/complex-bool/complex-complex/strong-0d-complex/
1-elem-array-sequence + 3D-inner-contig + negative-outer-stride). Full suite
green bar the 2 known pre-existing baseline fails. Leaf API (no internal Core
callers) so regression risk is isolated.
Implement ndarray.byteswap(inplace=False), a faithful port of NumPy's
PyArray_Byteswap (numpy/_core/src/multiarray/methods.c). Reverses the bytes
WITHIN each element to toggle the endian representation; the dtype is unchanged,
so only the reinterpreted values change. Complex swaps its real/imaginary halves
independently (NumPy copyswapn per component); 1-byte dtypes are a value no-op
(but inplace=False still returns a fresh copy). New files only — no existing
code touched.
Design
- byteswap depends ONLY on the element byte width, never on the dtype's numeric
meaning, so ONE width-dispatched SIMD kernel (swapUnit in {2,4,8,16}, complex
=itemsize/2=8) covers every dtype instead of a per-NPTypeCode path. VPSHUFB
reverses lane-locally, so one per-128-bit-lane mask drives AVX2 (32 B/iter) and
SSSE3 (16 B/iter) via GetLower(); scalar two-pointer swap for the tail /
no-SIMD fallback. NumSharp-only dtypes (no NumPy analog) take the consistent
raw-itemsize rule: Char swaps in 2s, Decimal in 16s.
- inplace=False = NumPy PyArray_NewCopy(ANYORDER) then swap. Fused into ONE pass
for a C-contiguous source (allocate uninitialized C-contiguous result, then
read->shuffle->write), skipping the copy machinery's second pass. F-contiguous
keeps NumPy's F layout via Shape.Order; 1-byte is a straight Buffer.MemoryCopy.
- Large (>=2 MiB), 32-byte-aligned fused writes stream non-temporally
(Avx.StoreAlignedNonTemporal + StoreFence) to skip the read-for-ownership a
fresh destination would otherwise pay (what Buffer.MemoryCopy avoids). Large
fresh buffers are page-aligned, so no alignment prefix is needed. NT does NOT
help the in-place path (the read already owns the line).
- inplace=True on a non-contiguous (strided/transposed/reversed) view drives a
single-operand EXTERNAL_LOOP + READWRITE + KEEPORDER NDIter whose kernel swaps
each chunk (unit-stride chunk -> SIMD block; else per-element) - NumPy's
IterAllButAxis inplace path. One-segment (C or F contiguous) swaps the whole
contiguous byte block wholesale. A non-writeable array (broadcast / read-only)
raises ValueError("array to be byte-swapped is read-only"), verbatim.
Verification
- 117-case in-process differential vs NumPy 2.4.2: 13 dtypes x 9 layouts
(1d / 2d / F-contig / strided / reversed / transposed-3d / 0-d / empty /
broadcast), both not-inplace and in-place (the strided in-place path exercises
the EXTERNAL_LOOP writeback), plus 13/13 broadcast -> ValueError. C# dumps
input+swap+inplace bytes; Python reconstructs the identical input via
frombuffer and validates. All bit-exact.
- Char/Decimal self-consistency (double-swap identity, raw byte reversal),
strided-inplace base-integrity (only viewed elements touched), read-only guard,
and large-array SIMD+non-temporal correctness.
- 23 unit tests (Casting/NDArray.byteswap.Test.cs), green on net8.0 and net10.0;
full Casting namespace (843) + FuzzMatrix gate green.
Perf (NPY/NS, best-of-21, Release, GC-clean)
- inplace 1.25-8.8x (only 10M int64/float64 at 1.39-1.49x, the DRAM-bandwidth
wall - both sides saturate memory, NumSharp the faster side).
- not-inplace 1.9-9.7x at every non-trivial size; sub-1.5x only at n=1000 fresh
allocation (the per-NDArray construction floor).
…tier)
Wire ndarray.byteswap into the NumPy 2.4.2 differential-fuzz gate. byteswap is a
dtype-preserving array-result op, so it lands in the manip tier alongside
flip/trim_zeros: gen_manip records the result's C-contiguous bytes (layout-
agnostic comparison), and char_tier("manip") weaves Char coverage automatically
via the uint16 proxy.
- gen_oracle.py: add ("byteswap", {}, lambda v: v.byteswap()) to gen_manip's base
job list. It is an ndarray METHOD (there is no np.byteswap), and not-inplace
never raises, so it needs no nd/sz guard and runs on every layout incl. 0-d,
empty, strided, reversed, transposed and broadcast (read-only -> copy).
- OpRegistry.cs: case "byteswap" -> ops[0].byteswap().
- manip.jsonl: regenerated (numpy==2.4.2). 364 new byteswap cases — all 13 NumPy
dtypes + Char, 26 layouts each. The bulk of the diff is id renumbering (case
ids carry a running counter), not semantic churn.
Gate: FuzzCorpusTests.Manip replays the whole tier (16019 cases) bit-exact vs
NumPy 2.4.2 — green, so every byteswap case (complex per-half swap, 1-byte
no-op, float NaN/inf byte patterns, every layout) is byte-identical. Decimal is
not part of a NumPy tier (no analog; byteswap there would be circular) and stays
on the unit-test self-consistency gate.
Follow-up to 27a19ae. NumSharp has no ndim cap (NumPy refuses >64 dims), so the inner-contiguous fill fast path removed its ndim guard and the odometer heap-allocates its coord array past a 64-deep stack budget (FiniteScan.cs pattern). This regression test fills a [3,1,...,1,4] view sliced [::2] at ndim=70 and 200 -- exercising the heap-odometer branch that ranks <=65 (stack path) never reach -- and asserts the same result NumPy produces at its max rank 64 (9,9,9,9,4,5,6,7,9,9,9,9), at ranks NumPy cannot even represent. Committed as a follow-up (not amended into 27a19ae) because a concurrent session had already stacked commits on top of the fill commit.
Wire ndarray.byteswap into the NumSharp-vs-NumPy op-matrix benchmark, joined on (op, dtype, N) like every other op. byteswap is the one O(N) byte-permutation in the manipulation suite (a fused read->VPSHUFB->write pass, non-temporal stores above an L2-scale threshold), unlike the O(1) view ops (flip/rot90/transpose). - C# ByteswapBenchmarks (Benchmarks/Manipulation/) — BenchmarkBase, float64 2-D arr, [Params(Medium, Large)] = 100K/10M, [Benchmark(Description="np.byteswap(a)")] => _arr2D.byteswap(). Matches the FlipRot suite convention. The *Benchmarks. Manipulation.* namespace filter auto-includes it — no orchestrator change. - NumPy twin in numpy_benchmark.py::run_manipulation_benchmarks — arr_2d.byteswap() as "np.byteswap" (float64). normalize_op_name folds both "np.byteswap(a)" and "np.byteswap" to "np.byteswap", so the merge joins them. Smoke-verified: BenchmarkDotNet --list flat discovers the method; the NumPy suite emits the byteswap row (0.039ms @ 100K float64); the join key normalizes identically on both sides. Measured at float64 to match the suite; per-dtype correctness across all 15 dtypes is the differential-fuzz oracle's job. Full measured numbers come from the post-release benchmark.yml run.
…harp parity)
NDAxisIter is the axis-reduction FALLBACK for non-contiguous / sliced / broadcast
arrays (var / std / cumsum / cumprod / all / any along an axis). Its state struct
embedded a `fixed long[64]` outer-dimension scratch and threw
NotSupportedException("NDAxisIter currently supports up to 64 dimensions.") for any
raw ndim > 64 -- a hard cap that contradicts NumSharp's unlimited-dims design (the
contiguous DirectILKernelGenerator axis path is already uncapped, and NDIter uses
NativeMemory for unlimited dims). Found while sweeping for other ndim caps after the
ndarray.fill fast-path cap; it was the one genuine structural one (einsum's nop>32 /
MaxDims=64 are intentional NumPy einsum parity; NDMemOverlap's <=32 all heap-fall-back).
Reachable and proven: np.all(np.arange(...).reshape([2,3,1,...,1,4]).T, axis=0) with
ndim=65 threw NotSupportedException, even though the array is mostly size-1 padding
(the check keyed off raw ndim, but the scratch only ever stores NON-unit outer dims).
Fix: NDAxisState's three `fixed long[64]` buffers become caller-provided `long*`
(OuterShapePtr / SourceOuterStridesPtr / DestinationOuterStridesPtr). Each of the four
entry points (ExecuteSameType, ReduceDouble, ReduceBool, ReduceNumeric) stackallocs the
scratch sized by ndim -- `stackalloc long[Math.Max(1, ndim)]`, exactly NDIter.Execution.cs's
idiom -- and CreateState/CreateReductionState fill it; the ndim>64 throw is gone. No fixed
cap, and the by-value state shrinks (3 pointers vs 1536 embedded bytes).
Verified: values bit-identical to NumPy 2.4.2 at ndim=64 (its max rank) for var/std/
cumsum/all/any on a non-contiguous transposed array; no throw and values matching the
low-rank equivalent at ndim 65/70/200/1000. Full suite green bar the 2 known baseline
fails (0 regressions across 13707 tests -- axis reductions are heavily covered).
…fety) Follow-up to 8f34e8f. That commit moved NDAxisState's outer scratch from an embedded `fixed long[64]` to raw `long*` pointers into caller-stackalloc'd (non-pinned) stack memory. A plain struct holding such pointers can be boxed, stored in a heap field, or captured by a closure/async frame — any of which would outlive the entry method's stack frame and leave the pointers dangling (the compiler does NOT lifetime-track raw pointers). Making NDAxisState a `ref struct` enforces the stack-only contract at compile time, exactly as Span<T> and NDIterRef do. All uses already comply (returned by value from Create{State,ReductionState}, local in the 4 entry methods, `ref` params to the loop helpers — no escape), so it compiles unchanged. Also dropped the now-purposeless [StructLayout(Sequential)] (it existed for the fixed arrays; the type is now pure stack, never marshalled). Behavior identical (ref-struct is a compile-time constraint); axis reductions green.
…arity) Implement NumPy's byte-field view methods on NDArray, byte-exact with NumPy 2.4.2 across every layout, dtype and offset (152-case dtype-matrix differential + 34 unit tests, both TFMs green; verified in an isolated HEAD worktree amid concurrent Core churn, committed only-my-files-by-path). getfield(dtype, offset=0) returns a byte-reinterpreting VIEW: it reads `dtype` out of the `offset`-th byte(s) of each element while keeping EVERY byte-stride and dimension verbatim (unlike view(dtype), which rescales the last axis). So a complex128 array's real/imag parts are getfield(f64, 0) / getfield(f64, 8), and an int32's low/high halves are getfield(i16, 0) / getfield(i16, 2) — the narrow field of a contiguous array is a strided view that SHARES memory (writes through, unless the source is read-only). setfield(val, dtype, offset=0) is getfield + an assignment: it writes `val` (unsafe-cast, broadcast) into that field, leaving the rest of each element intact; returns void, in place. Implementation (2 new Core files + 1 test file, no existing code touched): - UnmanagedStorage.GetFieldAlias<T>(offset) / (NPTypeCode, offset) — the storage half, modelled on AliasComplexLane/WrapReinterpreted. NumPy carries strides in bytes; NumSharp in ELEMENTS, so each stride is recomputed as oldByteStride/newItemsize — always exact, because every NumSharp itemsize (1/2/4/8/16) divides every larger one, so an old byte-stride is always a whole multiple of the new. The one representation gap is a sub-element byte OFFSET (e.g. int32.getfield(i16, 1) starts mid-element, which element-granular Shape.offset cannot express): the remainder r = offset % newItemsize is absorbed into the WRAP POINTER (shifted r bytes) while Shape.offset carries the aligned part — so EVERY in-range offset is representable, matching NumPy's raw byte pointer. Contiguity flags recompute from the resulting strides (a same- size field of a C-contiguous array stays contiguous, a narrower one does not); a read-only source yields a read-only field view. Validation is NumPy's PyArray_GetField verbatim texts/order: "new type is larger than original type" / "offset is negative" / "new type plus offset is larger than original type". - NDArray.getfield / NDArray.setfield — the instance methods (Type / NPTypeCode / generic<T> overloads, matching the view() convention). setfield checks SELF writeable FIRST (ValueError "assignment destination is read-only"), before the dtype/offset validation, exactly as PyArray_SetField does; the value is coerced like fill_diagonal (scalar -> NDArray.Scalar, else asanyarray) and written via np.copyto(..., casting:"unsafe") (float->int truncates toward zero). Parity verified: complex real/imag, int low/high halves, all three misaligned byte-offset cases (incl. combined with a non-zero base offset from a sliced source), same-size reinterpret, strided/F-contig/transposed/negative-stride/ sliced sources, 0-d, empty, read-only (view stays read-only; setfield raises), all 15 field dtypes dispatched (Char/Decimal/Complex have no NumPy analog but plumb correctly by size). Sole divergence is the broadcast-mismatch error TYPE (setfield of a wrong-shaped array -> IncorrectShapeException vs NumPy's ValueError) — the pre-existing library-wide house convention inherited from copyto/broadcast_to, not specific to this feature.
…arity)
Implements ndarray.setflags(write=, align=, uic=) — the port of NumPy's
array_setflags (methods.c) — and fixes the three gaps the existing
NDArrayFlags object had against NumPy, every behavior probed live on 2.4.2.
setflags semantics (all probed, messages verbatim):
- Processing order align -> uic -> write, with NumPy's flagback ROLLBACK:
an error (uic=True, or a refused write=True) restores the entry state,
so a same-call align change never survives a failed call.
- write=True follows _IsWriteable (common.c), evaluated UNCONDITIONALLY
(a still-writeable view whose base went read-only is refused and stays
writeable): owner of ordinary memory -> always; view -> iff its base is
writeable; foreign read-only memory -> never. So np.broadcast_to views
CAN be re-enabled (writes then alias across stride-0 axes and reach the
source, matching NumPy exactly), while an 'r' memmap and a frombuffer
over a read-only buffer refuse with ValueError('cannot set WRITEABLE
flag to True of this array').
- align=False observably clears ALIGNED (flags.aligned/num/behaved/
carray/farray and the repr all follow; num 1287->1031 on owned 1-D);
align=True restores. NumPy's mis-aligned refusal is unreachable here
(element strides/offsets - data can never sit mis-aligned), documented.
- uic=True raises verbatim; uic=False is a deliberate no-op divergence:
NumPy also SEVERS the view's base (v.base -> None, data can dangle) -
NumSharp's base chain roots the owner for the view's lifetime, and
detaching it would be a use-after-free with no writeback state to
resolve ([Misaligned]-pinned).
flags gaps closed:
- NDArrayFlags.writeable setter now routes through setflags (as NumPy's
arrayflags setters literally call arr.setflags), replacing the old
refuse-all-views rule with the real _IsWriteable rule. flags['W'/'A'/
'X'] setters route the same way, so flags['A']=false is now LIVE
(aligned was a hardcoded const true with a no-op setter).
- ALIGNED reads live from Shape flags; fresh views/copies of an
align-cleared array re-report aligned=True (NumPy recomputes alignment
per new array object): re-set at the _Allocate funnel (covers GetData's
identity-subshape and sliced-view storages) and in the verbatim-shape
alias factories (Alias/Alias(Shape)/Alias(ref)/WrapReinterpreted).
Write-protection plumbing (the memory-safety half of _IsWriteable):
- UnmanagedStorage.WriteProtected marks storages wrapping foreign
read-only memory; CanEnableWriteable() is the _IsWriteable analog.
- NpyFormat memmap 'r': the WRAP storage (the owner) is now marked AND
shape-cleared BEFORE the F-order reshape/transpose views are taken -
previously only the FINAL view's shape was cleared, leaving the wrap
storage writeable underneath; with the new base-chain re-enable rule
that hole would have let a write reach PROT_READ pages (access
violation, not an exception). The empty-'r' branch is marked too
(NumPy refuses setflags(write=True) on an empty 'r' memmap).
- np.frombuffer over a read-only MemoryView marks the boundary storage,
so the direct re-enable refuses like NumPy's writable-buffer probe;
memmap 'r+' round-trips (writable buffer), 'c' stays writeable.
Perf (NPY/NS, Release, best-of-9): setflags write pair 9.9x (21.9 vs
217.9 ns), align pair 7.5x, no-op call ~94x, flags.writeable/num reads
30-36x, 3-D write pair 10.6x - metadata ops with no Python dispatch.
Gates: test/NumSharp.Tests/Backends/NDArray.setflags.Test.cs (23 tests -
owned/view/broadcast/memmap r+rF+r+/frombuffer/0-d/rollback/one-call/
bracket-routing, incl. 2 [Misaligned] divergence pins), green on net8.0
+ net10.0; full suite 13759 passed with only the 2 known pre-existing
baseline failures (T1_33 Char name, NoOperationDefersItsBufferRelease);
FuzzMatrix oracle 78/78 (Alias/_Allocate are hot paths - bit-exactness
unchanged). setflags is a metadata protocol with no (dtype,shape,bytes)
product, so it is unit-test-gated like nditer/ndindex, not fuzzed.
…flags + 5 owndata/error parity fixes Adds the flags oracle — the heavy coverage gate that pins ndarray.flags and ndarray.setflags to NumPy 2.4.2 across the whole layout/producer space, plus the five parity fixes building it exposed. Zero excused divergences: all 626 corpus cases replay bit-exact. The oracle (house differential pattern — committed corpus, no Python in CI): - test/oracle/gen_flags_oracle.py probes LIVE NumPy 2.4.2 over 53 recipes × ~9 setflags transition scenarios + a 13-dtype x 6-layout independence sweep, emitting Backends/corpus/flags_oracle.jsonl (626 cases, 118 error cases). Recipes cover: owned C/F 1-5D, 0-d + 0-d view, empty + empty sliced, transposed 2D/3D, strided, negative-stride 1D/2D, offset/step/ composed slices, row/col views, newaxis, broadcast_to (full/same/scalar/ partial), broadcast_arrays, fancy 1D/2D, boolean mask, reshape view/copy, ravel view/copy, view(dtype) same/diff itemsize, diag/diagonal read-only views, imag(real), real(complex), astype, copy C/F, eye, frombuffer ro/rw, memmap r / r+ / c / F-order / empty. - Each case pins the ENTIRE flags record (c_contiguous, f_contiguous, owndata, writeable, aligned, writebackifcopy, fnc, forc, behaved, carray, farray, num) + the verbatim 6-line str(flags) on base cases + verbatim ValueError type/message on error cases + the POST-ERROR state (NumPy's flagback rollback). num is masked with 0x7FFFFFFF: broadcast_arrays results carry NumPy's internal WARN_ON_WRITE bit (0x80000000) inside num, deprecation machinery NumSharp deliberately does not model (same reason bcast_arrays0 skips the str compare - NumPy renders 'WRITEABLE : True (with WARN_ON_WRITE=True)'). - FlagsOracleTests.cs replays every case through FlagsOracleRecipes (a 1:1 C# twin of the Python builder) and adds NumPy-free heavy sweeps: all 22 bracket keys vs their dotted twins across every recipe (+KeyError get/set), a 15-API write-guard enforcement sweep across the setflags toggle (indexer/ slice/fancy/mask sets, fill, copyto, ufunc out=, put, place, sort, partition, byteswap, shuffle, flatiter), the flags-equality==num-equality law over all 53x53 recipe pairs + hashcode, np.require O/W interplay (memmap copies under 'O'), corpus floors (53 recipes, >=600 cases, >=100 errors, every op token present), a comparator self-test (perturbed expectations must be detected), and the [Misaligned] pin for the one known divergence (slice of a setflags-writeable broadcast reverts to read-only: NumSharp recomputes broadcast W on fresh views; a NumPy-style blanket inherit would silently undo np.diagonal-class deliberate-clear contracts). Parity fixes the oracle exposed (each probed against NumPy 2.4.2): - fancy-index results now report owndata=True (NumPy parity): the gather built the owned buffer then handed back dst.MakeGeneric<T>() / a terminal dst.reshape(retShape) - a pure type-plumbing ALIAS whose _baseStorage chained to the invisible fresh temp, misreporting flags.owndata=False and base!=null. The flat lane now allocates at retShape directly and FetchIndicesND returns the owned instance (a caller-supplied @out still gets the typed wrapper - a genuine view of the user's array). - UnmanagedStorage.ExternalBase + OwnsData: memmap arrays report owndata=False in EVERY mmap_mode (NumPy: the mmap object is the base) - previously the wrap storage had no base so owndata read True; 'r+'/'c' stayed re-enable-able (writable buffer), 'r' stays WriteProtected. np.require('O') now copies memmaps (reads OwnsData). - np.imag(real) returns its owned read-only zeros directly (owndata=True, base=None in NumPy) instead of a read-only ALIAS of them (owndata=False); setflags(write=True) on it now succeeds as NumPy's _IsWriteable allows. - empty slices are views: a[100:200] used to allocate a fresh owner (owndata=True); NumPy reports owndata=False with the base set. The empty storage now chains _baseStorage, which also gives the NumPy answer for setflags(write=True)-after-parent-readonly. - in-place sort on a read-only array raises NumPy's verbatim ValueError('sort array is read-only') - was InvalidOperationException with invented text (partition/byteswap already carried NumPy's exact texts). Gates: FlagsOracleTests 12/12 green on net8.0+net10.0 (626 corpus cases, ~7.5K pinned fields); full suite 13771 passed with only the 2 known pre-existing baseline failures; FuzzMatrix oracle 78/78 (fancy/empty-slice changes are hot indexing paths - bit-exactness unchanged). Regenerate the corpus with: python test/oracle/gen_flags_oracle.py (numpy==2.4.2).
…ses) + 6 parity fixes + np.isfortran Scanned NumPy 2.4.2's OWN source for every flags consumer/producer (PyArray_FailUnlessWriteable call sites across multiarray/umath; .flags reads across numpy/lib, _core, linalg) and turned the inventory into a second oracle wave: the corpus grows 626 -> 871 cases (76 recipes, 170 error cases), still ZERO excused divergences, plus the parity gaps the scan itself exposed - each probed live before fixing. Scan-driven corpus additions (test/oracle/gen_flags_oracle.py): - 23 new producer recipes: squeeze C/F, swapaxes 3D, expand_dims, atleast_2d, rot90, flip, mT, split/unstack children, getfield(i32@0 of i64), pad C/F, delete F, insert F, concatenate CC/FF (the F-vote), zeros_like(F) / ones_like(T) / empty_like(strided) (order='K' preservation), meshgrid(copy=False) broadcast views, and CROSS-OBJECT states (view of a read-only owner, view(dtype) of a read-only owner) - each with the full ~9 setflags transition scenarios, so the _IsWriteable matrix is now corpus-pinned across every producer. - NEW identity-vs-copy consumer section: ascontiguousarray (C identity / F copy / strided copy), asfortranarray (F identity / C copy), ravel(C-1D view) - pins result flags AND the shares-memory verdict (np.shares_memory on the NumPy side; a write-probe on the C# side). - meshgrid_nocopy joins bcast_arrays0 in skipping the str compare (NumPy renders 'True (with WARN_ON_WRITE=True)' - unmodeled warn machinery). Parity fixes the scan exposed (all probed against NumPy 2.4.2): - np.squeeze is now a stride-dropping VIEW (PyArray_Squeeze): dims AND strides drop, offset/buffer stay. The old reshape-based squeeze lost F-contiguity (F input came back C - the issue-#610 gap), MATERIALIZED non-contiguous inputs where NumPy shares memory, and turned a broadcast squeeze into a writeable copy (a documented divergence in WriteabilityMatrixTests, now RESOLVED to parity and re-pinned as such). - np.delete / np.insert allocate in the SOURCE's order - NumPy's arrorder = 'F' if arr.flags.fnc else 'C' (_function_base_impl.py): F input -> F-contiguous owned result (num 1286). Implemented as an IDEMPOTENT relabel (WithSourceOrder) at every worker return, safe under nested overload delegation. - ndarray.resize ownership check reads OwnsData, not IsView - a memmap ('r+') resize used to SUCCEED and would have reallocated mapped memory; now refuses with NumPy's exact text ('cannot resize this array: it does not own its data'). NumPy checks OWNERSHIP only - a read-only OWNER resizes fine (probed; pinned). - np.nditer with a readwrite/writeonly op-flag on a read-only operand refuses at construction - nditer_constr.c:1068 verbatim ('operand array with iterator write flag set is read-only'); used to silently construct. - flatiter assignment names its target like NumPy: 'underlying array is read-only' (iterators.c:754) - was the generic 'assignment destination'; guarded at Scatter entry too so even a zero-length fancy set refuses. - NEW API np.isfortran(a) == a.flags.fnc (numpy/_core/numeric.py) - was missing entirely. New NumPy-free law sweeps in FlagsOracleTests (over the corpus-pinned fields): np.isfortran == fnc across every recipe; memoryview(a).readonly == !writeable across every C-contiguous recipe (floor-guarded against vacuity); real/imag setters refuse read-only ('assignment destination is read-only', getset.c:807); the resize ownership matrix (memmap/view refuse, read-only owner allowed); the nditer write-flag guard; comparator self-test extended to detect a wrong shares-memory expectation. Gates: FlagsOracleTests 18/18 green net8.0+net10.0 (871 corpus cases bit-exact, ~10K pinned fields); full suite 13777 passed with only the 2 known pre-existing baseline failures; FuzzMatrix oracle 78/78 (squeeze is used throughout - bit-exactness unchanged). Regenerate: python test/oracle/gen_flags_oracle.py (numpy==2.4.2).
…nds (split contiguity, fft out= guard, out= texts)
A paired live-probe sweep (NumPy 2.4.2 vs NumSharp, ~130 records over the
surfaces waves 1-2 did not pin: producers applied to READ-ONLY sources,
cross-object _IsWriteable chains, ALIGNED recomputation, the bracket-key
setter matrix, and out= write-guards beyond the pinned 15 APIs) uncovered
three problem kinds. Each is fixed and corpus-pinned; everything else in
the sweep matched bit-for-bit.
Kind 1 — np.split children mis-derived contiguity (silent wrong flags):
SplitContext.DeriveSubFlags inferred the child's C/F flags ALGEBRAICALLY
from the parent's flags (parentC && (axis==0 || sameLen), mirrored for F),
which cannot express NumPy's size-1 contiguity relaxation:
- a (1,4) child of a C-only (3,4) parent is BOTH C- and F-contiguous in
NumPy (num 259) - the collapsed axis stops mattering; NumSharp said F0;
- a (1,6) parent split on axis 1 keeps its (1,3) children C-contiguous
(leading singletons void the axis-position rule);
- a broadcast ENDS when the split axis collapses to length 1
(split(broadcast_to(a,(4,3)), 4)[0] is C1F1 W0 num 259, not BROADCASTED).
DeriveSubFlags now takes the built child dims and recomputes broadcastness
+ contiguity with the canonical Shape walks (ComputeIsBroadcastedStatic /
ComputeContiguousFlagsStatic, made internal - the exact
_UpdateContiguousFlags port the walking ctor uses), so the fast path can
never drift from the canonical computation. np.split was the ONLY
view-producer deriving flags for the no-walk Shape ctor (unstack et al.
use the walking ctor). Verified over a 31-case singleton-dim matrix
(slices/T/F/3-D/broadcast/empty x split/indexer) - all C/F flags now match
NumPy; the remaining diffs are singleton-axis STRIDE VALUES only
(semantically arbitrary; NumPy keeps parent strides, NumSharp normalizes).
Kind 2 — np.fft out= silently wrote into a read-only array:
np.fft.fft(a, out=ro) validated only the shape, then the driver wrote the
buffer directly - corrupting a setflags(write=False) target, a broadcast
view, or an 'r' memmap. One guard in RawFft covers the whole family
(fft/ifft/rfft/irfft + hfft/ihfft compositions), placed AFTER the shape
check to match NumPy's probed order (_raw_fft's Python-level shape check
fires before the ufunc's writeable validation): wrong shape -> ValueError
"output array has wrong shape."; read-only -> "output array is read-only".
Kind 3 — out= write-guard texts and validation ORDER (probed per API):
- cumsum/cumprod/clip/matmul: read-only refuses FIRST - its message wins
even when the shape is ALSO wrong - with NumPy's ufunc wording "output
array is read-only" (was the generic "assignment destination is
read-only" raised late from copyto, after computing the whole result).
- take/compress: shape error first ("output array does not match result
of ndarray.take"), safe-cast second, read-only LAST with take's own
quirky verbatim text "WRITEBACKIFCOPY base is read-only"
(PyArray_TakeFrom wraps out in a NPY_ARRAY_WRITEBACKIFCOPY view, so its
writeability check names the writeback base).
- cumsum/cumprod wrong-size text is now NumPy's verbatim "provided out is
the wrong size for the accumulation." (type stays the house
IncorrectShapeException per the long-standing shape-error convention).
All raised through NumSharpException.ThrowIfNotWriteable(shape, name) -
the same house type + verbatim-text pattern the elementwise ufunc out=
plumbing already uses.
Corpus grows 871 -> 998 cases (87 recipes, 199 error cases - 174 uic +
25 writeable), still ZERO excused divergences:
- +11 recipes: the four split children (split2d_row/split2d_ax1/
split_f3d_ax0/split_bcast_end - the fixed bug class), imag_complex (the
missing writeable imag-of-complex VIEW sibling of real_complex), and six
read-only-source producers (ro_T, ro_owner_T, ro_reshape_copy, ro_fancy,
ro_ascontig, ro_astype_nocopy) pinning inheritance vs copy-resets-W vs
identity-preserves-RO. The ro_T/ro_owner_T pair pins a subtle NumPy
rule: arange().astype().reshape() is a reshape-VIEW, so w1 on its
read-only transpose SUCCEEDS (the collapsed base chain skips the
read-only intermediate and lands on the writeable astype owner); only
the true-owner source (np.zeros) refuses.
- +14 chain/ cases (new corpus section 5 + FlagsOracleRecipes.BuildChain
twin): _IsWriteable re-evaluated after the base's own state changed
(view_w1_after_owner_reenabled), re-enable through view-of-view chains
(subview_of_ro_view_w1), flag NON-propagation to existing views
(existing_view_owner_w0/a0; existing_view_w1_refused pins that the
refusal leaves the view writeable), ALIGNED recomputation on fresh
views/copies of an align-cleared owner, and the w0->w1 toggles on
memmap r+/r/c and frombuffer rw/ro views.
New NumPy-free gates in FlagsOracleTests:
- OutGuards_ReadOnlyOut_NumpyVerbatimTexts_AndOrder: the kind-2/3 family -
exact texts, per-API validation order, and re-enabled targets accepting
every previously-refused call.
- SplitChildren_FlagsEqualEquivalentSliceViews: the anti-drift law - every
split child's flags.num equals its slice-built twin's across 11 layouts
(C/F 2-D+3-D, transposed, leading-singleton, broadcast end/keep/ax1,
read-only source).
- Comparator self-test extended to detect a perturbed chain expectation;
floors raised (>=990 cases, ==87 recipes, >=190 errors, chains >=14).
Gates: FlagsOracleTests 21/21 green net8.0+net10.0; np.split suite 72/72;
FuzzMatrix oracle 78/78 (split is used throughout - bit-exactness
unchanged); full suite 13780 passed with only the 2 known pre-existing
baseline failures. Regenerate: python test/oracle/gen_flags_oracle.py
(numpy==2.4.2).
…') and ones/zeros order= parity fixes
A paired live-probe sweep (NumPy 2.4.2 vs NumSharp, ~128 records) over the
RESULT-flags surfaces the single-array recipe catalog never reaches:
order= producers (copy/ravel/flatten/astype x C/F/A/K x C/F/strided
source), stacking/joining, axis reductions, creation-with-layout, and
advanced indexing. It uncovered two genuine parity bugs and seven
numpy-internal-representation divergences.
Two parity fixes (both probed against NumPy 2.4.2):
- ndarray.astype(order='C') on an F-contiguous source was IGNORED. The
order handling only had the 'F' half (force an F-copy when the resolved
order is F but the cast result is not F-contiguous); there was no
symmetric 'C' branch, and TensorEngine.Cast preserves the source layout
(a same-dtype copy of an F array stays F), so F.astype(t, order='C')
silently kept F where NumPy produces a C-contiguous result. Extracted a
shared RelayoutAstype that imposes BOTH orders; every other combo
(c_F, f_F, f_K, f_A, all _A/_K) already matched, so the fix is the one
missing case. Guards the empty sentinel (new NDArray(typecode), null
dimensions) by reading the raw dims field — Shape.NDim/IsContiguous NRE
on a null-dims shape, and the original only touched Shape inside the 'F'
branch (which never fired for the sentinel); pinned by the pre-existing
CastEmptyNDArray test.
- np.ones(shape, order=) / np.zeros(shape, order=) did not exist (only
np.empty had the overload). Added both, mirroring np.empty — the fill is
order-independent, so the requested layout is observable only through the
flags, which is the whole point of the parameter.
New corpus section — the layout matrix (74 cases, base flags only, all
bit-exact with NumPy):
- order= producers: copy/ravel/flatten/astype x {C,F,A,K} x {C,F,strided}
source (pins the astype fix directly and the whole #610 order surface).
- creation-with-layout: zeros_F/ones_F/empty_F/eye_F (pins the new order
overloads), eye(k=1), identity, tri, triu, tril, diagflat, full,
full_like(order='F'), ascontiguousarray(F)/asfortranarray(C).
- axis-reduction / compute results: sum/mean keepdims, cumsum(axis) on
C and F inputs, diff, sort(C), argsort, matmul (C and F), where, clip,
unique.
- advanced indexing: fancy rows, fancy elements, boolean 2-D mask,
take_along_axis, choose, ix_.
Emitted as layout/{kind}.{name} with a C# BuildLayout twin whose keys are
identical to the Python builder; excluded from the recipe-iterating law
sweeps (they name producer expressions, not catalog recipes).
Seven numpy-internal-representation divergences — documented, NOT pinned in
the bit-exact corpus (VALUES identical in every case, only layout FLAGS
differ), captured by the new [Misaligned] test
LayoutDivergences_NumpyInternalRepresentations_Documented:
1. A full reduction / np.trace returns a read-only numpy SCALAR (num=263);
NumSharp has no scalar type, so its 0-d result is a writeable array.
2. reshape(order='F') that must reorder COPIES in NumSharp (owndata=True);
NumPy returns a view of an internal F-copy (owndata=False).
3. A strided reshape NumPy expresses as a non-contiguous VIEW, NumSharp
materialises as a C-contiguous owned copy.
4. np.stack of F operands: NumPy's layout vote yields neither-contig,
NumSharp yields F-contig.
5. np.sort of an F array returns a C-contig copy (house convention, already
in CLAUDE.md); NumPy keeps F-order.
6. np.nonzero returns contiguous OWNED index arrays; NumPy returns strided
views into one shared (ndim, count) buffer.
7. np.linspace owns its buffer; NumPy's is astype(copy=False) over an
internal array (owndata=False).
Corpus grows 998 -> 1072 cases (still 87 producer recipes, 199 error
cases); new gate Corpus_LayoutMatrix_MatchNumpy (floor 70), floors raised
(>=1060 cases). FlagsOracleTests 23/23 green net8.0+net10.0; FuzzMatrix
78/78 (astype/ones/zeros used throughout — bit-exactness unchanged); full
suite 13782 passed with only the 2 known pre-existing baseline failures.
Regenerate: python test/oracle/gen_flags_oracle.py (numpy==2.4.2).
…contiguousarray/asfortranarray (close consumer-surface gap)
Audit of 'is np.MemoryView (ndarray.data) accessible in every API NumPy uses/exposes
memoryview in' found the PRODUCER side complete (ndarray.data; numpy scalar .data maps
to 0-d NDArray.data) and one CONSUMER covered (np.frombuffer(MemoryView)), but the
array-like consumer surface was NOT reachable with a MemoryView.
In NumPy a memoryview is array_like via the buffer protocol (ctors.c PyMemoryView_FromObject),
so np.array (copy), np.asarray/asanyarray/ascontiguousarray/asfortranarray (view) and the
np.ndarray(buffer=) constructor all accept it, PRESERVING the buffer's N-D shape/layout
(unlike frombuffer's always-1-D reinterpret). NumSharp's MemoryView is a bespoke handle,
not array_like, so those calls did not bind:
- np.array/asarray/ascontiguousarray/asfortranarray(mv) were COMPILE ERRORS
- np.asanyarray(mv) compiled (bound to 'in object') but threw NotSupportedException
Fix: one MemoryView overload per function, each a one-line forward to the existing NDArray
overload via buffer.obj (the source array) — so np.X(mv) == np.X(mv.obj), inheriting the
already-tested copy/view/layout/dtype semantics. The asanyarray overload is more specific
than the 'in object' converter, so it wins resolution. asarray gets Type + string-dtype
overloads. np.ndarray(buffer=) needs no new API — np.frombuffer is its analog.
All semantics probed against NumPy 2.4.2:
- np.asarray(a.data) -> zero-copy view, shares memory, preserves 2-D shape
- np.array(a.data) -> independent copy, preserves shape; writeable even from a
read-only (broadcast) source
- np.ascontiguousarray(F.data) -> C-contiguous; np.asfortranarray(C.data) -> F-contiguous
- np.asarray(a.data, float32) / (a.data, "float32") -> dtype convert
Gate: 8 new round-trip tests in APIs/np.memoryview.Test.cs (25 total, net8+net10) pinning
view-vs-copy write-through, N-D shape preservation, C/F forcing, dtype convert,
readonly-broadcast->writeable copy, and null-argument throws. Purely additive (205 insertions,
0 deletions); full-solution build clean (no overload-resolution regressions, nothing converts
implicitly to MemoryView); 157 adjacent frombuffer/asarray/asanyarray/array tests green.
…ntations (NumPy 2.4.2 parity)
Closes every divergence the flags-oracle waves left documented as [Misaligned]: the
values were already identical everywhere; these were the remaining flag/strides/
shares-memory observables where NumPy exposes an internal representation NumSharp
did not model. All probed against NumPy 2.4.2; each item below is corpus-pinned by
32 new `parity.` layout cases in flags_oracle.jsonl (generated from REAL NumPy, now
1104 cases; layout floor 70->100, total floor 1060->1100) plus rewritten POSITIVE
unit pins (LayoutParity_NumpyInternalRepresentations_Modelled,
SliceOfWriteableBroadcast_InheritsTheOverride — both formerly [Misaligned]).
1) reshape = port of _reshape_with_copy_arg + _attempt_nocopy_reshape (shape.c):
- NEW Shape.TryNocopyReshape (line-for-line port, element units; size-1 old axes
dropped, trailing size-1 new axes take next-fastest stride; negative and zero
strides ride the arithmetic) + NDArray.ReshapeCore routing all reshape overloads.
- Same-dims (checked BEFORE -1 resolution) -> PyArray_View full alias.
- Contiguous-in-order (or size<=1/0) -> relabel view at the SAME offset.
- Nocopy success -> (possibly non-contiguous) strided VIEW sharing memory:
arange(24).reshape(3,8)[:, ::2].reshape(2,6) is the byte-stride (96,16) view
(write-through), a reversed 1-D splits to negative strides (-32,-8), broadcast
stride-0 axes split to stride-0 runs (read-only inherit).
- Else copy in the requested order and return a VIEW of that internal copy
(owndata=False on the copy path too — exactly NumPy's base semantics), which
also completes reshape(order:'F'): view of an internal F-copy, -1 now allowed.
- order 'A' = PyArray_ISFORTRAN; order 'K' = ValueError verbatim.
- NDArray.flat pins its documented materialize-on-non-contiguous contract
explicitly (its ~15 consumers walk the buffer linearly); ravel/flatten
unchanged (owned copies, matching NumPy's PyArray_Flatten).
2) np.sort/np.partition = copy(order='K') + in-place (fromnumeric verbatim): F input
keeps F (8,24), 3-D transpose keeps its exact stride order (8,96,32) num=1284,
strided/negative come back C, broadcast comes back F-ish (8,32).
3) True KEEPORDER allocation: NEW View/StridePerm.cs ports
PyArray_CreateSortedStridePerm + BuildStrides; NDIter.CopyAs 'K' now allocates
along the sorted stride perm for neither-contiguous sources (astype/copy K of a
3-D transpose / broadcast match NumPy exactly). astype() no longer re-imposes a
binary-resolved order over Cast's KEEPORDER result (this also stops
astype(copy:false) forcing copies on satisfied strided sources).
np.ravel_multi_index materializes via an explicit order:'C' astype (its IL kernel
walks linearly; NumPy's own C uses PyArray_ContiguousFromAny).
4) concatenate/stack output layout = full PyArray_CreateMultiSortedStridePerm vote
(ported verbatim in StridePerm.MultiSortedPerm: size-1 axes never vote, C wins
conflicts, only-set-while-ambiguous). stack-of-F yields NumPy's neither-contig
strides — (96,8,24)/(16,8,48)/(16,48,8) for axis 0/1/2 — while concat of F/T
inputs stays F and both-C&F columns stay C. All-C fast path answers in O(ndim)
because the insertion sort cannot exit early through ambiguity: np.r_'s uncapped
ndmin=100000 padding (all size-1 axes) measured 12 s through the vote, and NumPy
never sees this only because it caps ndim at 64.
5) np.nonzero returns NumPy's representation: ONE shared (count, ndim) C-order
multi-index buffer (filled by the argwhere row-expand IL kernel — one sequential
write stream) whose COLUMNS are the tuple entries: strided views (byte stride
ndim*8, owndata=False, num=1280; 1-D collapses to a unit-stride view of the
(count,1) base, num=1283; empty keeps the stride pattern). np.where(cond)
inherits. Consumers (fancy indexing, trim_zeros, ix_) handle the strided views.
6) np.linspace float64 reports owndata=False (NumPy's scalar path mutates the
arange-reshape VIEW in place and astype(copy=False) returns it verbatim);
num<=1-with-endpoint, dtype casts, and num=0 own — including the probed
num=1/endpoint=False view case.
7) Full reductions return NumPy's read-only SCALAR flags (PyArray_Return): NEW
NDArray.MarkReductionScalar clears WRITEABLE on fresh 0-d results (num=263) at
every reduction exit — sum/prod/mean/amax/amin/std/var/nan*/argmax/argmin
(elementwise + shared axis executors, incl. 1-D-axis-reduced-to-0-d),
all/any, trace (fast + general), median/percentile/quantile (QuantileEngine),
ptp. keepdims arrays and out= operands stay writeable; keepdims over a 0-d
input is still a scalar (probed). Residue: all/any results ride MakeGeneric's
alias so their 0-d owndata bit reads 0 where NumPy's scalar owns (pre-existing
generic-wrapper class, unpinned).
8) Views of a setflags(write=True)-re-enabled broadcast INHERIT the override
(slice/step/T/squeeze all num=1280, matching NumPy's inherit-the-bit rule):
the Alias funnels re-set WRITEABLE when the parent is an already-broadcast,
explicitly re-enabled array and the child's computed False came only from the
inherited broadcastness. Plain broadcast views stay read-only; broadcast_to
still clears explicitly (even over a re-enabled broadcast parent).
Gates: flags oracle 23/23 (1104-case corpus regenerated from numpy 2.4.2);
FuzzMatrix + index oracle + metamorphic 150/150 on net8.0+net10.0; full unit suite
13,790/13,803 on both TFMs (the 2 remaining failures are the documented
pre-existing baselines: NoOperationDefersItsBufferRelease, T1_33 Char dtype-name).
Known doc staleness (not touched here — .claude/CLAUDE.md is dirty with parallel-
session WIP): the partition "house np.sort convention (C-contiguous copy)" note,
the concatenate "F only when all-F and not all-C" vote note, and "flat COPIES via
reshape" should be refreshed to match this commit.
…ts for the modelled representations — exposes and fixes 2 missed reduction-scalar exits Heavy differential + integration coverage for 53b5d82 (the 8 formerly-[Misaligned] numpy-internal representations), in the house oracle pattern: a committed corpus generated from REAL NumPy 2.4.2, replayed bit-exact in C# with no Python in CI. NEW test/oracle/gen_layout_parity_oracle.py -> Backends/corpus/layout_parity_oracle.jsonl (210 cases, 14 families) + LayoutParityOracleTests (15 family gates + floors). Unlike the flags oracle (flags records only), every case records shape, BYTE STRIDES, flags.num/ owndata/writeable, an exact shares-memory verdict against the case's source, and the result VALUES as base64(tobytes(order='C')) — compared bit-for-bit. C#-side the replay additionally runs write-through probes (a shares=1 view must mutate its source; a shares=0 result must not), asserts nonzero's tuple entries alias ONE buffer (InternalArray identity), and checks partition's two-sided anchor invariant per lane: - reshape (65): every _reshape_with_copy_arg route — same-shape view, C/F/A relabel, nocopy combine/split over positive/negative/zero strides and offset windows (5-D, 0-d, empty, broadcast splits), copy-in-order fallbacks — incl. a 13-dtype sweep of the flagship strided nocopy-view cell. - sort (22) / argsort (4) / partition (4): KEEPORDER layouts x {C,F,T,strided,negrow, broadcast,t3d,offset} x axes incl. flat, 6-dtype sweep, NaN lane (layout + NaN-last pinned; value bytes corpus-excluded — NumPy canonicalizes sorted NaN to 0x7ff8..., NumSharp's radix to .NET's 0xfff8..., the long-documented set-ops-era divergence). - stack (9) / concat (13): the multi-operand stride vote incl. neither-contig stacks, mixed dtype, mixed contiguity, 3 inputs, axis=None flatten, empties, both-C&F columns. - nonzero (19) + where1/argwhere/flatnonzero: shared-buffer column views across layouts and dtypes (argwhere/flatnonzero pin VALUES only — their layout rides the documented pre-existing base-of-transpose/owning-engine residue). - copyk (10): copy/astype order='K' over t3d/broadcast/strided/negrow/F. - linspace (10): view-vs-owning matrix incl. num=1/endpoint=False (i64 cell pins layout only — NumPy floors the float lattice where Converts.ToInt64 rounds half-even, a pre-existing linspace int-dtype value divergence outside this scope). - reduce (41): the read-only scalar contract (num=263) across sum/prod/mean/std/var/ amin/amax/median/ptp/trace/argmax/percentile/quantile/nan* x int64/float64, the 1-D-axis and keepdims-over-0-d scalar cells, keepdims-array and out= writeable contrasts, all/any writeable-only rows. - bcastw (9): the writeable-override inheritance chains + plain/rebroadcast controls. THE COVERAGE FOUND 2 REAL GAPS (both now fixed and corpus-pinned): 1) The np-layer nan statistics kept writeable 0-d results: nanmean/nanstd/nanvar's own NDArray.Scalar exits (and nanmean's 0-d-input Clone path) live in Statistics/np.nan*.cs, outside the engine files the original marking pass swept. All 14 exits now route through MarkReductionScalar. 2) HandleScalarReduction — the shared size<=1 early path behind sum/amax/amin/prod — builds its result via Cast/Clone (not NDArray.Scalar), so the original pass missed its exit: np.sum(0-d, keepdims=True) and every size-1-input full reduction returned writeable where NumPy returns the read-only scalar. Its exit is now marked. NEW LayoutParityIntegrationTests (34 tests) proves the semantics compose with the rest of the codebase: reshape nocopy-views write through 4-deep view chains and feed reductions/sort/fancy-index/mask/astype/printing identically to materialized copies, serve as fancy-index operands, receive copyto, inherit read-onlyness from broadcast/ diagonal sources, refuse resize on the copy path, and survive source-wrapper GC (ARC); a 15-dtype write-through sweep (Char/Decimal included); K-sort outputs satisfy take_along_axis(argsort)==sort on exotic layouts, are writeable, idempotent under re-sort, and Decimal/Char ride the same K machinery; stack's neither-contig outputs equal their C-contiguous twins through sum/reshape/T/print, accept slice writes, and unstack round-trips as write-through views; nonzero views drive fancy get/SET (clearing exactly the nonzero cells), mutate only the shared buffer (disjoint columns proven), and agree with count_nonzero across layouts and all 15 dtypes; read-only reduction scalars refuse SetAtIndex/copyto/fill with NumPy's verbatim "read-only" error yet work everywhere as operands (arithmetic, broadcasting, reduce-of-scalar, copy/astype back to writeable); the broadcast override lands stride-0 writes in the shared base cells, never leaks into fresh broadcasts, and re-clearing re-protects new views. Gates: LayoutParityOracleTests 15/15 + LayoutParityIntegrationTests 34/34; full suite 13,839/13,852 on net8.0 AND net10.0 (the 2 remaining are the documented pre-existing baselines); oracle project 150/150. Corpus regeneration: python test/oracle/gen_layout_parity_oracle.py (asserts numpy==2.4.2).
…l struct (was the dedicated StridePerm class) The KEEPORDER stride-permutation ports (PyArray_CreateSortedStridePerm, PyArray_CreateMultiSortedStridePerm, the perm-stride builder, the KEEPORDER allocation shape and the concatenate output-layout vote) are pure dimension/ stride computations — the concern the Shape struct owns, and where NumPy itself keeps them (numpy/_core/src/multiarray/shape.c, beside the reshape machinery already ported as Shape.TryNocopyReshape). They now live on the Shape partial (View/Shape.StridePerm.cs, the house per-concern-partial pattern next to Shape.Reshaping/Shape.Broadcasting) instead of a dedicated static class: StridePerm.SortedPerm(strides) -> Shape.SortedStridePerm(strides) StridePerm.BuildStrides(dims, perm) -> Shape.StridesForPerm(dims, perm) StridePerm.MultiSortedPerm(shapes, n) -> Shape.MultiSortedStridePerm(shapes, n) StridePerm.KeepOrderShape(src) -> src.KeepOrder() (instance) StridePerm.ConcatOutputShape(...) -> Shape.ConcatOutputShape(...) Pure relocation — algorithm bodies, comments (verbatim-port quirks, the all-C O(ndim) fast path for np.r_'s uncapped ndmin) and both call sites' semantics (NDIter.CopyAs order='K', np.concatenate's output allocation) are unchanged. NumPy parity re-proven by the untouched differential gates: layout_parity oracle 15/15 + integration 34/34 + flags oracle 23/23 on net8.0 AND net10.0, full suite 13,839/13,852 (the 2 documented pre-existing baselines), oracle project 150/150.
Refactors benchmark documentation to separate guidance from generated output: `benchmark/README.md` is now a concise orientation guide, while `benchmark-report.md`/`history/latest` are emphasized as canonical results and the DocFX dashboard is called out as the human-facing UI. The benchmark docs and skill references were updated with the NPY/NS convention, credibility/"negligible" gating, artifact ownership, and CI publishing behavior. Also fixes stale type-count wording from 12 to 15 in C# benchmark docs, updates snapshot manifest card wording, and adds `choose` to the Selection API list in `.claude/CLAUDE.md`.
Renamed the solution folder from "oracle" to "NumSharp.Tests.Oracle.Python" so it matches the Python-based oracle tooling and is clearer in Solution Explorer and the solution tree.
…PORDER allocation (issue #610) Axis reductions (sum/mean/prod/min/max/std/var + all nan* variants) flipped an F-contiguous input to a C-contiguous result, where NumPy preserves the layout. Root cause: the reduction executors allocated the output C-contiguous unconditionally, and the keepdims step reshaped through a fresh C-shape which also reset the strides to C. NumPy does NOT copy to fix this — its reduce iterator allocates the output operand in KEEPORDER and the reduction loop writes directly into that (already F-strided) buffer. NumSharp now does exactly the same, with no post-hoc copy: - New DefaultEngine.AllocateReductionResult(outputType, outputDims, inputShape) allocates the result in OrderResolver.Resolve('K', inputShape) order — F for a strictly-F-contiguous input, C for C-contiguous / general-strided (matching NumPy for pure-C and pure-F; the same C/F-only 'K' model cumsum, np.copy and empty_like already use). AllocateReductionZeros is the zero-filled sibling for the degenerate size-1-reduced-axis std/var paths. - keepdims now re-inserts the reduced axis with Storage.ExpandDimension(axis), which preserves the layout, instead of Storage.Reshape(new Shape(ks)), which reset it to C. Every output-writing mechanism already honours the result's strides, so filling an F-strided buffer needs no copy and no kernel change: the IL axis kernels write via outputStrides (their C-contiguous fast paths are gated on a C-contiguous INPUT, so an F input never reaches them), the scalar and nan-scalar paths via SetAtIndex -> Shape.TransformOffset, the np-layer nan loops via SetDouble(coords), and NDAxisIter via dstOuterStrides. argmax/argmin are deliberately excluded — NumPy allocates their index output C-order regardless of input layout (probed 2.4.2). Converted executors: ExecuteAxisReduction, ExecuteAxisReductionNDIter, HandleTrivialAxisReduction (Add), ExecuteAxisStdReductionIL + fallback + trivial (Std), the Var equivalents, ExecuteNanAxisReduction + ExecuteNanAxisReductionScalar (Nan), and the hand-rolled np.nan{mean,std,var} axis paths. Verification: 0 flag divergences vs NumPy across 392 C/F x op x axis x keepdims cases (incl. the size-1-axis edge), 0 value mismatches across 312 F-vs-C checks (all 13 ops), FuzzMatrix 98/98 bit-exact, 1592 statistics/reduction tests green. Known residual (documented, not a regression): a genuinely transposed neither-C-nor-F input reduces to a C result where NumPy's full stride-sort KEEPORDER may pick F — values are byte-correct, only the flag differs, and it is invisible to the oracle (which serializes ascontiguousarray on both sides). Tests: the six F-contig-reduction OpenBugs tests in OrderSupport.OpenBugs.Tests now pass and are un-marked ([TestCategory("Fixed")]) so they run in CI as regression gates. The two adjacent stale placeholders Flip_ApiGap and Repeat_FContig_Axis0_ApiGap (hardcoded false.Should().BeTrue() reminders left over from before np.flip / np.repeat(axis) shipped) are rewritten as real tests.
…enBugs "BUG 1", add byte-exact str+repr regression tests
NDArray.ToString on a broadcast view whose SOURCE was itself sliced
(reversed / stepped / sliced-column / slice-of-slice) is already BYTE-EXACT
with NumPy 2.4.2 on journey3 — the array_str/array_repr layout engine walks
the array in logical C-order through Shape.TransformOffset (dimension-based
GetCoordinates -> GetOffset with the real strides), so a stride=0 broadcast
axis stacked on non-trivial view strides (negative / >1 / row-major-column /
compound double-slice) resolves correctly. Verified against NumPy 2.4.2 across
21 broadcast-view variations (reversed, stepped, sliced-column, double-sliced,
transposed+sliced, negative-2D, 3D, newaxis, F-contiguous source, 0-d-indexed
row, scalar, summarized-2000/30-wide, negative-row, and dtypes int/float/half/
bool/complex) — every one byte-identical for BOTH str() and repr().
The four OpenBugs "BUG 1a-1d" (Bug_ToString_ReversedSliceBroadcast /
StepSliceBroadcast / SlicedColumnBroadcast / DoubleSlicedBroadcast) documented
a value bug that no longer exists AND asserted the WRONG form: they called
ToString(false) (= np.array_str, SPACE separators -> "[[2 1 0]...]") while
asserting .Contain("2, 1, 0") (the COMMA form, which is repr()/array_repr).
They were therefore red purely on a separator mismatch, not a value defect.
Resolution (the class doc's "move passing tests to the permanent test class"):
- OpenBugs.cs: remove the 4 stale/redundant tests and the obsolete "BUG 1"
comment block; leave a resolved-note breadcrumb pointing at the permanent
regression tests.
- np.ArrayPrint.ParityTests.cs: add region "broadcast-of-a-view" with 6 tests
asserting BYTE-EXACT str AND repr for the four scenarios plus the 3D and
transposed+sliced combinations. This closes a real coverage GAP — the str
form was already covered in np.ToString.BattleTests.cs (region "Broadcast +
Slice Combinations"), but there was NO repr regression test for broadcast+
slice views anywhere.
Net: no core change (the feature already has full NumPy parity); the stale
OpenBugs become permanent, correct, CI-running (non-excluded) regression tests.
np_ArrayPrint_ParityTests 186/186 green (net8.0 + net10.0), validated in an
isolated detached worktree (the shared working tree is transiently non-compiling
from an unrelated parallel-session edit to NonContiguousTests.cs).
…uction layout (issue #610) Adds ReductionKeepOrderTests (40 tests) pinning the F-contiguity-preservation cases the OrderSupport OpenBugs set does not reach. Every expected value/flag is NumPy 2.4.2's exact output. - Every VALUE op (prod/min/max/std/var) and every NaN op (nanprod/nanmin/nanmax/ nanmean/nanstd/nanvar, incl. the complex np-layer path) preserves F on a 3-D F-contig input — one per distinct executor. - RARE: reducing a SIZE-1 axis of an F-contig array still keeps F (the HandleTrivialAxisReduction / AllocateReductionZeros path — the last gap closed). - keepdims at each axis position (first/middle/last) — the last-axis case appends the size-1 axis via ExpandDimension and must stay F. - Collapse-to-<=1-non-unit-dim results (leading/middle/trailing unit dim, 1-D result, empty input) report BOTH C- and F-contig, matching NumPy's flag rule. - Dtype coverage: int32 (widen->int64, Direct path), float32/float64/Complex/ Decimal (NDIter), Half (float32-shadow NDIter) all preserve F with the correct output dtype. - Values are correct despite the F layout: F-result equals C-result across all value + nan ops (np.array_equal), plus exact NumPy spot values. - Regression guards: C-contig input stays C (not over-applied); argmax/argmin stay C-order (NumPy allocates their index output C-order regardless of input). - Documented residual [Misaligned]: a genuinely transposed (neither-C-nor-F) input reduces to a C result where NumPy's full stride-sort KEEPORDER picks F — the VALUES are byte-correct (6, 86 match NumPy), only the flag differs, consistent with the codebase's C/F-only 'K' model. - ddof != 0 std/var on an F buffer (the in-place scale post-pass). All 40 green.
…creation/ctor fixes
Adds CreationCtorEdgeCaseLiveParityTests (14 methods) alongside the happy-path
CreationCtorLiveParityTests — the boundary and degenerate inputs the first suite does
not reach, every case byte-compared against embedded CPython numpy 2.4.2. Verified live
on net8.0 + net10.0: 14/14 passed, 0 skipped (both interop classes together: 22/22).
T1.44 (F-order ctor):
- size-1 dims (1,6)/(6,1)/(1,1,6)/(2,1,3)/(6,1,1) — numpy flags these BOTH C- and
F-contiguous; NumSharp matched flag-for-flag and element-stride-for-stride.
- empty (0,3) (0 bytes, F-contig), 1-D (F==C), and 4-D higher-rank F.
- all 11 numeric dtypes (i1..u8, f2/f4/f8) byte-exact.
- F-strided WRITE lands at the correct physical slot (write [1,0]=99 -> compared to
numpy's post-write array), and the sibling (IArraySlice, Shape, 'F') ctor.
T1.50 (arange bool): negative-step (2,0,-1), fractional-step (0,1,0.5)/(0,2,1.5),
empty and single-arg overload byte-exact for length<=2; length>2 raises on BOTH sides
with NumSharp's TypeError text asserted equal to numpy's captured message.
T1.56 (ndmin): ndmin=-1 (no-op) / 0 / 5 (prepend four 1s), a 2-D input at ndmin 0/3,
and an EMPTY input at ndmin 2 ((1,0)) and 0 ((0,)) — shapes and bytes match numpy.
T1.54 (frombuffer): count / offset / count+offset / empty byte-exact (via numpy slices);
all 13 supported dtype spellings (?,i1..u8,f2/f4/f8,c16) round-trip byte-exact; big-endian
'>i2/>i4/>i8/>u2/>f8' byte-swap to native (values match numpy's native-cast reference —
a documented dtype divergence); and size%itemsize!=0 raises ArgumentException with numpy's
verbatim "buffer size must be a multiple of element size" text (numpy raises ValueError).
T1.47 (promotion): find_common_type over 8 same-kind / int->float combos equals numpy's
modern result_type live; _can_coerce_all(start=0/1/2) equals result_type of the tail,
start==length gives Empty, and the List<> sibling overload with start>0 no longer throws.
The suite self-skips (Inconclusive) where Python+numpy is absent and is x64-reference per
InteropTestBase; these creation cases carry no cross-arch rounding, so they stay strict
byte-exact on every host with the engine.
…ases (issue #610) Follow-up to the ReductionKeepOrderTests set, adding 4 more rare cases: - Negative axis (-1, -2) — normalized before the reduction, then KEEPORDER applies, so an F-contig input stays F. - 4-D F-contig single-axis reduction — bridges the 3-D and 6-D cases, confirming the fix is rank-agnostic. - An OFFSET F-contig input (a column slice, offset != 0, still F-contig): the reduction reads through Shape.offset and yields NumPy's exact values (22, 26). All 44 ReductionKeepOrderTests green.
…tests (str, repr, array2string) Broadens the broadcast-of-a-view printing coverage begun in d56d31f with 42 more regression tests in np_ArrayPrint_ParityTests, each asserting BYTE-EXACT parity with NumPy 2.4.2. Every expected string was produced by running the equivalent NumPy 2.4.2 code and then differentially cross-checked in-process against NumSharp (a 45-case NumPy-vs-NumSharp sweep, all byte-identical for both str() and repr()) before being embedded. New regions: - all 13 NumPy dtypes on broadcast-of-a-reversed-view (str + repr): bool, int8, uint8, int16, uint16, int32, uint32, int64, uint64, float16, float32, float64, complex128 -- pins the dtype= repr suffix rule and the per-dtype element formatter selection through the stride=0 + negative-stride combination. - summarization (threshold): reversed/stepped 1-D broadcast to (2,1000), a 50-row column broadcast, a 3-D (2,3,100), and a both-axes (60,60) -- the leading/trailing edgeitems Walk/Recurse paths read through view strides. - line wrapping at linewidth (wide int and float broadcast rows). - special values: nan/inf, signed zero, complex nan/inf, float32 nan. - non-broadcast view repr parity (reversed/stepped/transposed/column/both- reversed across int64/float64/float32/complex128/bool) -- closes a gap where only str() of plain views was pinned (np_ToString_BattleTests). - higher rank & inserted axes: newaxis-in-the-middle, 4-D, 5-D. - np.array2string options over broadcast views: threshold, edgeitems, precision, sign='+', separator=', ', suppress_small. No core change (the feature already has full NumPy parity). Printing suite now 228/228 green per TFM (net8.0 + net10.0), validated in an isolated detached worktree at HEAD (the shared working tree is transiently non-compiling from an unrelated parallel-session edit to NonContiguousTests.cs).
…uous bool Follow-up to the np.clip-Boolean-strided fix — adds NumPy-2.4.2-verified tests for the rarer layouts and bound shapes the two graduated regression tests did not reach. Every value/shape/dtype was probed against NumPy 2.4.2 first (21 cases, all bit-exact) before being pinned. np.clip.Test.cs (8 new): - Clip_Bool_Strided_ArrayBounds — per-element ARRAY bounds on a strided view producing a NON-constant result ([T,F,T,F] clipped by lo=[F,T,F,T] hi=[T,T,F,F] -> [T,T,F,F]); the strongest check that source AND strided-paired bounds resolve the right C-order slots (a constant lo=hi=True result would hide a read-mapping bug). - Clip_Bool_BroadcastBound_Transposed — a (1,)-shaped bound broadcasts (stride=0) across a transposed view. - Clip_Bool_MinGreaterThanMax_AllFalse — lo=True > hi=False saturates to False (Max-then-Min order) through transposed and strided views. - Clip_Bool_MinOnly_MaxOnly_Transposed — the two single-bound modes on a transpose. - Clip_Bool_3D_Transposed_Identity — higher-rank (4,2,3) multi-axis strided read. - Clip_Bool_NegativeStride_2D_Identity — doubly-reversed [::-1,::-1] view. - Clip_Bool_InPlace_StridedOut — out= aliasing a strided view: the write lands back at the ::2 positions of the base, odd positions untouched. - Clip_Bool_Empty_And_Singleton_NonContiguous — empty transposed + 1-element strided. NonContiguousTests.cs (4 new): - InvertBoolean_Transposed_AllForms — ~, np.invert, np.logical_not, np.bitwise_not agree on a transposed (F-contiguous) view. - InvertBoolean_3D_Transposed — higher-rank flip. - InvertBoolean_BroadcastView_ReturnsFreshWriteable — ~ of a read-only broadcast (stride=0) view yields a fresh WRITEABLE C-contiguous array, not a view. - NegativeBoolean_UnaryMinusOperator_AlsoRejected — the unary `-` operator (not just np.negative) throws NumPy's boolean-negative TypeError, including on a broadcast view. All 16 green on net8.0 and net10.0.
…y 2.4.2)
np.clip on a bool array threw "NotSupportedException: clip not supported for
Boolean" on the general strided/transposed/F-order/reversed path while the
contiguous path worked — the coordinate-iteration kernel (DefaultEngine.
ClipStrided) omitted the Boolean case in its dtype switch, falling through to
`default: throw`. NumPy 2.4.2 clips bool at every layout (clip(bool,True,True)
-> all-True bool; clip(bool,False,True) -> identity).
Fix: Boolean now rides the existing Byte Min/Max kernel in ClipStrided
(ClipStridedT<byte> with &Math.Max/&Math.Min). bool storage is a single 0/1
byte (the contiguous IL path already loads it via Ldind_U1 and selects on the
0/1 value), so unsigned Byte ordering reproduces false<true exactly — the
strided result is bit-identical to the contiguous 0/1 scalar select, for scalar
AND array (incl. broadcast stride=0) bounds. No new per-dtype path, no struct
kernel: it reuses the Byte case's kernel verbatim. All 15 dtypes are now handled
on the strided clip path (bool was the only gap).
Also corrects two stale test expectations: NumPy removed the boolean-negative
loop, so np.negative(bool)/unary `-` is now a TypeError and `~` (np.invert /
logical_not) is the flip. NumSharp already mirrored both; the two
NonContiguousTests.NegateBoolean_* pins asserted the pre-2.x int result and were
rewritten to the 2.4.2 contract (regression + edge tests land in the sibling
test commit).
Test bookkeeping:
- Removed the two [OpenBugs] pins Clip_Bool_{Transposed,Strided}_Throws from
OpenBugs.DtypeCoverage.cs (the file's own "remove the carve + test when fixed"
convention / iscomplex precedent); the named regression + edge tests live in
np.clip.Test.cs.
- Un-carved bool in gen_oracle.py CLIP_DTYPES (bounds = (False, True) so clip is
an identity, gating that the strided read maps to the right C-order slot),
regenerated stat.jsonl (+7 bool clip cases across every STAT layout, numeric
data unchanged). FuzzMatrix stat tier green on net8.0 and net10.0.
…able + drop int-era growth cap Fixes the two [OpenBugs] tests HashsetLongIndexingTests.HashHelpersLong_GetPrime and HashHelpersLong_ExpandPrime_ProgressiveGrowthTest, both annotated "HashHelpersLong.primes array contains non-prime values". HashHelpersLong is the long-indexed port of .NET's System.Collections.HashHelpers that backs Hashset<T> (consumed by ConcurrentHashset<T> and np.unique's hash path). Its capacity table must be all-prime: prime bucket counts are what keep the modulo hash distribution collision-free. Two defects broke that contract. Root cause 1 — composite table entries. The 47-entry "Extended primes for large collections" section (indices 72..118, values ~8.6e6 .. ~37.9e9) was generated as a ~1.2x geometric progression whose members were never verified prime: 35 of the 47 are composite (e.g. 8639243 = 7*1234177, 10581492263, 37915399963). GetPrime returns the first table entry >= min, so GetPrime(1e10) returned the composite 10581492263 and the test's IsPrime(result) assertion failed. (In DEBUG this also tripped the latent Contract.Assert(IsPrime(newSize)) in SetCapacity for those sizes.) Fix: each composite bumped up to nextprime(value). Values stay >= the original (so the ">= min" and growth guarantees hold) and strictly increasing; the 84 already-prime entries — including the base-72 block and the two the IsPrime test pins (7199369, 4252464407) — are byte-identical. Root cause 2 — the MaxPrimeArrayLength cap. It was 0x7FFFFFC7 (~2.147e9), the 32-bit Array.MaxLength copied verbatim from the int-based original and never adjusted for the long retrofit. In ExpandPrime it fires when oldSize < cap < newSize, so an oldSize of 2e9 (past the 1e9 large-growth threshold) capped to ~2.46e9 instead of reaching its 33%-growth target of 2.666e9 — failing ProgressiveGrowthTest. It also wasn't prime. The table already returns up to ~38e9 for larger oldSize (e.g. 5e9 -> 7.3e9), proving the ceiling was meant to track the table, not the 32-bit array limit. Fix: MaxPrimeArrayLength = 37915399967L, the largest table entry (now prime). Beyond the table GetPrime computes as before. Validation: both target tests plus all 28 HashsetLongIndexingTests pass (incl. the 1M -element stress test, which drives the real Initialize/IncreaseCapacity/ExpandPrime/ GetPrime/SetCapacity resize path). 133 unique + Hashset/ConcurrentHashset consumer tests pass — the corrected bucket sizing changes collision behavior only, never the set of values a consumer produces. Verified against the compiled NumSharp code that all 119 entries are prime, strictly increasing, and MaxPrimeArrayLength is prime.
…closed
The ArrayFlags.OWNDATA bit (0x0004) was declared since the flags enum landed but
never set by any code path, so Shape.OwnsData and (Shape.Flags & OWNDATA) always
read false — while the REAL ownership tracking lived only on UnmanagedStorage
(_baseStorage is null && !ExternalBase). The two AuditV2 OpenBugs pinning this
(T1_29_OWNDATA_Flag_Never_Set, T1_64_Flags_OWNDATA_Always_False) now pass and
run in CI as regression pins.
Design — mirror NumPy's array-object flags word (probed against the source):
numpy ctors.c PyArray_NewFromDescr_int raises NPY_ARRAY_OWNDATA when IT
allocates (line 923) and clears it when data is passed in (line 931), and
common.c's _IsWriteable asserts base != NULL => !OWNDATA. Shape._flags is
already NumSharp's array-object flags word (WRITEABLE and ALIGNED are
maintained statefully through the same funnels), so OWNDATA joins them as the
third stateful flag, kept in lockstep with the storage truth:
- UnmanagedStorage.SyncOwnDataFlag() (new): reconciles _shape's OWNDATA bit
with OwnsData. Called at every funnel where ownership or _shape transitions:
* _Allocate (allocating constructors set the bit),
* the ~30 codegen'd scalar/vector ctors (via pre-owned ScalarOwnedShape /
OwnedVectorShape statics — zero per-construction cost; codegen template
comments updated in lockstep),
* every _baseStorage assignment (all 3 Alias overloads, WrapReinterpreted,
AliasComplexLane, CreateBroadcastedUnsafe, 4x GetData view overloads,
GetView's empty+contiguous branches, GetFieldAlias) — a view borrows the
parent's shape struct, which carries the parent's bit and must drop it,
* ExternalBase raises in NpyFormat (memmap 'r'/'r+'/'c'; 'r+' takes no later
shape swap, so the explicit sync is the only thing clearing the wrap bit),
* SetShapeUnsafe / ExpandDimension / the ReplaceData family — in-place shape
swaps assign freshly built shapes that never carry the bit, which would
otherwise silently strip it from an owner (NumPy: in-place mutation never
changes OWNDATA; methods.c re-enables it after in-place resize).
- NDArrayFlags.O now reads _arr.Shape.OwnsData (was Storage.OwnsData), so the
1104-case flags oracle + 210-case layout-parity oracle gate the MIRROR: any
future funnel that drops or fails to set the bit turns the corpus red
instead of leaving Shape.OwnsData silently wrong. np.require keeps reading
the storage truth; tests assert the two never disagree.
- Shape.WithFlags now routes through the no-walk 7-arg ctor: dims/strides are
unchanged so size/hash are carried verbatim instead of recomputed — the
pre-existing WRITEABLE/ALIGNED fixups in the alias hot paths lose two
O(ndim) walks per call, more than paying for the new OWNDATA reconcile.
Verified against live NumPy 2.4.2 on an ~80-case probe matrix (owndata + base
+ flags.num for creation/views/copies/in-place/memmap): NumSharp's storage
truth already matched NumPy on all rows except the 5 KNOWN open op-level
ownership divergences (np.squeeze no-op-on-owner, np.resize function,
np.flatnonzero, np.argwhere, trailing-advanced-axis fancy X[:,[i,j]]) — those
are op-semantics work, unchanged here and excluded from the new tests. After
the fix the C# twin probe reports Shape bit == Storage truth == flags surface
on every row with every flags.num unchanged.
Tests:
- AuditV2_LogicShapeStorage: T1.29/T1.64 de-[OpenBugs]'d into regression pins.
- NEW Backends/OwnDataFlagTests.cs (26 tests, every expectation live-probed
NumPy 2.4.2): allocating creation incl. 0-d/empty/per-dtype ctor families;
copy-returning ops (flatten, non-contig ravel, fancy, mask, sort, unique,
concatenate, take, repeat, diag-construct, arithmetic, axis/keepdims
reductions); view families (slices incl. empty+full-slice Alias borrow,
view()/view(dtype)/getfield, axis movers incl. identity rollaxis + 1-D
transpose, reshape views AND the reshape-copy path returning a
view-of-internal-copy owndata=False, expand_dims/squeeze, broadcast_to +
broadcast_arrays, diagonal/diag-extract, real/imag complex lanes, flip,
rot90, split children, tile, roll, nonzero/where shared-buffer columns);
in-place preservation (resize, storage reshape, ExpandDimension,
ReplaceData, setflags write/align both ways incl. num=263/259 pins);
np.imag(real) read-only OWNER (num=263); memmap never owns in any mode;
a 34-case three-way mirror-agreement invariant sweep; Shape-struct
semantics (hand-built shapes + the SHARED Shape.Scalar static carry no
ownership claim, WithFlags OWNDATA roundtrip preserves size/hash/refs,
the bit does not perturb Shape equality); pinned NumPy flags.num
composites (1287/1285/1283/1282/1281/1280/256).
- Drive-by: Diagonal_AllDtypes_Smoke repaired — its Boolean row used
np.arange(9, bool), which the arange-bool guard from 5c7e3ad (correctly,
NumPy TypeError past length 2) now rejects; the bool 3x3 comes from astype.
Gates: full suite 14424/14424 green on net8.0 AND net10.0 (CI filter);
FuzzMatrix 98/98; flags + layout-parity oracles 72/72.
The OWNDATA reconcile introduced in f46bd21 reads better as the storage's post-transition lifecycle hook: it runs after _shape is (re)assigned or the ownership links (_baseStorage / ExternalBase) change, and reconciles the shape's OWNDATA bit with UnmanagedStorage.OwnsData. Pure rename — all 39 occurrences across the 10 core files + 2 test files (call sites, XML crefs in Shape.cs, comments); the method doc's opening now anchors the hook name. No behavior change: full suite 14424/14424 (net10.0, CI filter), flags + layout-parity oracles and OwnDataFlagTests green.
…lock, packed kernel key, direct EXLOOP advance, SIMD comparison out=, eager overlap-temp dispose NDIter's per-call setup tax was the dominant cost of every NDIter-routed op below ~10K elements, and the ufunc out= route was slower than NumPy at n=1 on every op. Measured (Release, `dotnet run -c Release` probes vs NumPy 2.4.2 in the same session): three-operand 1-D construction 161 ns, of which 3 calloc/free pairs = 78 ns per-call kernel-cache key string 45 ns + 96 B garbage (4 enum formats + concat) Broadcast + ResolveUfuncIterationShape 55 ns + 2 Shape allocations (identity work) per-chunk driver (iternext + dispatch) 5.6 ns np.less(f64, f64, out=bool) @100k 43.4 us vs NumPy 11.4 (scalar-only inner loop) np.add(a, b, out=a) (COPY_IF_OVERLAP) forced-copy temp leaked to the finalizer per call Six changes, all behaviour-preserving (14,446/14,446 main suite, FuzzMatrix 98/98 bit-exact, net8.0 + net10.0 build): 1. Single-block state + per-thread recycling (NDIter.State.cs, NDIter.cs). NDIterRef.AllocateStateBlock allocates the header and a 1536-byte inline arena in ONE native block; AllocateDimArrays carves the dimension and per-operand arrays from the arena (falls back to the separate blocks when ndim x nop outgrows it; stack states from CreateCopyState/CreateReductionState have no arena and keep the old path). Dispose / FreeState / Copy's failure path return the block through ReleaseStateBlock, which parks it in a bounded [ThreadStatic] 4-slot cache after ResetForRecycle re-zeroes the header and only the carved bytes — every array the iterator hands out must start zeroed (BaseOffsets is +=-accumulated by FlipNegativeStrides; Coords/Buffers/BufStrides too). Construction: New 1-op 146 -> 62 ns, 3-op EXTERNAL_LOOP 161 -> 93, ufunc flags 197 -> 113. AdvancedNew's catch now tears down through FreeState (also frees buffers a failed Initialize had allocated). 2. Packed kernel-cache key (Backends/Kernels/InnerLoopKernelKey.cs, InnerLoop.cs). InnerLoopKernelKey packs family/op/dtypes into one ulong; a struct-keyed front cache (_innerLoopKeyCache) sits over the string-keyed _innerLoopCache, and ToCacheKey() reproduces the legacy "npy_binop_{op}_{l}_{r}_{res}" string byte-exactly on a miss, so DynamicMethod names, kernel identity and GeneratedDelegates.InnerLoopCount are unchanged (ClearInnerLoop clears both). NDIterRef gains packed-key ExecuteElementWise / ExecuteElementWiseBinary / ExecuteElementWiseUnary overloads whose cache hit allocates nothing; the nine production sites (BinaryOp, CompareOp, UnaryOp x2, UfuncOut x4, Shift) use them. The string overloads stay for user kernels (Tier 3A/3B/3C). 3. Direct EXTERNAL_LOOP advance (NDIter.Execution.cs, NDIter.cs). ForEach, ExecuteGenericMulti and ExecuteReducing call ExternalLoopNext directly (inlined, state array pointers hoisted into locals so the JIT stops reloading them per operand per axis) instead of through the cached NDIterNextFunc delegate when IsPlainExternalLoopAdvance() — the production ufunc configuration. Per-chunk driver overhead 5.6 -> 4.7 ns on 4-element rows; buffered and non-EXLOOP iterators keep the delegate path untouched. 4. SIMD comparison for out= (NDIter.Execution.cs TryExecuteComparison, UfuncOut.cs). ExecuteComparisonUfuncInto compiled a scalar-only Tier-3B body for EVERY layout. When the inputs share a dtype and the bool out is contiguous in iteration order it now runs the whole-array SIMD comparison kernel (Vector.Compare + mask packing — the kernel the no-out route already uses for contiguous inputs) through the iterator's post-coalesce strides; mixed dtypes, cast outs and strided/F outs keep the scalar body. np.less(f64, f64, out=) @100k: 43.4 -> 9.3 us (NumPy 11.4). 5. Same-dims fast path on the three *UfuncInto routes (UfuncOut.cs). When lhs/rhs/out have identical dims the input broadcast and the out join are identity transforms that cannot raise (Shape.Equals compares dims only), so Broadcast + ResolveUfuncIterationShape are skipped: -55 ns and two Shape allocations on np.add(a, b, out=o). 6. Eager COPY_IF_OVERLAP temp dispose (NDIter.cs ResolveWritebacks, NDIter.Detach.cs ResolveDetachedWritebacks). After the write-back the operand slot reverts to the user's original (NumPy's operands revert once WRITEBACKIFCOPY resolves) and the forced-copy temp is disposed instead of dropped to the finalizer — one leaked pooled buffer per in-place ufunc call before. inplace np.add(a, b, out=a) @1: 282 -> 154 ns (NumPy 570). Net at n=1 (ns, NumSharp before -> after | NumPy 2.4.2): add(out=) 411 -> 168 | 289, less(out=) 366 -> 140 | 300, sqrt(out=) 338 -> 183 | 258, negative(out=) 329 -> 153 | 267, positive(out=) 332 -> 141 | 257; add(out=) @1k 510 -> 233 | 438; less(out=) @1k 794 -> 232 | 417. Managed garbage per add(out=) call 552 -> 200 B. Every out= op measured is now faster than NumPy at n=1 (all were slower). Official nditer benchmark sections (NPY/NS, fresh both sides): construction geomean 2.73x -> 5.8x (ctor.3op_exl 8.6x, ctor.ufunc 9.3x), lessbool 0.71/0.49/0.18/0.60x -> 2.07/1.82/1.25/1.06x across 1/1K/100K/1M, inplace@1 0.62x -> 3.7x, zerodim 0.62x -> 3.2x, chunkwidth w=4 0.48x -> 0.82x, w=16 1.02x -> 1.13x. Benchmark findings recorded in benchmark/nditer/README.md and CLAUDE.md: - nditer_sheet.py now runs the C# side with DOTNET_TC_CallCountingDelayMs=0: tiered compilation only promotes after a 100 ms window with no new tier-0 JIT, and a benchmark process keeps jitting DynamicMethods for most of a 200 ms row, so the first rows of every section were timed at tier-0 (lessbool@1 484 ns default vs 124 ns with the knob; astype@1 755 vs 346; the identical call read 130 ns as a later row of the same process). NumPy has no JIT, so this only ever inflated the NumSharp side of the scalar/1K tiers. - The committed 2026-08-29 sheet's 1M/10M elementwise cells (add@10M 0.33x, sqrt@10M 0.23x, mixbuf@10M 0.50x) do not reproduce: re-run in isolation they read 0.85x / 0.97x / parity, and BOTH sides' committed 1M/10M numbers were 2-5x slower than a fresh run (host contamination at snapshot time). Ceilings left (measured, documented): narrow-row per-chunk cost is now the kernel delegate call plus the odometer (~4.5 ns) and NumPy's np.positive does 5.9 ns/row on 4-wide rows via its buffered transfer + one vectorized loop — beating that is a kernel-selection matter (2-D strided whole-array kernels, which production np.positive already takes), not iterator overhead; the raw V256 add at 1M is 0.85x NumPy with no iterator at all (kernel/alignment level). Tests: NDIterStateBlockTests (8 — cache park/pop, bounded cache, recycled block hosting a different iterator incl. the FlipNegativeStrides BaseOffsets hazard, arena overflow fallback, Copy, Detach/FreeState, production route across layouts), InnerLoopKernelKeyTests (4 — legacy string format byte-exact, field round trip incl. NPTypeCode.Complex=128, 21x15x15 + 6x15x15 + 41x15x15 + 2x15x15 identities collision-free, production route registers under both keys), NDIterComparisonOutRouteTests (10 — all 15 dtypes, scalar/row broadcast, strided/reversed inputs, strided and F-order out fallbacks, mixed-dtype common-type semantics, NaN, out aliasing an input, cast out).
The command taught 'dotnet run <<'EOFDOTNET'' / 'python <<'EOFPYTHON'' as the general probe/validate/benchmark mechanism with no size caveat. A large or quote-dense script inlined in a heredoc is emitted as one giant single-command string that can corrupt in transport and fail with 'unexpected EOF while looking for matching '' (the doc/JSON/prose-dense case is the worst). Added a note: for large/quote-dense scripts, write the script to a file (probe.cs / probe.py) and redirect it in (dotnet run -c Release - < probe.cs / python probe.py); short snippets can stay inline. Aligns with the benchmark skill, which already mandates file-based timing scripts (dotnet run -c Release - < script.cs, fresh filename per rebuild). Small python twins (measure-verify / O1_EXCLUSIONS) are legitimate small heredocs and were left as-is.
…s behind it docs/NDITER_PERF_DISCOVERY.md is the developer-facing write-up of the 2026-09-03 NDIter performance pass (commit 15154b0): the cost model (FIXED + rows x PER_CHUNK + elements x PER_ELEMENT, with the measured decomposition of a 411 ns np.add(out=) call), the ten measuring traps with the evidence that each produced a wrong number first, the probes, the code map of the six levers, the full before/after result tables (construction vs np.nditer, the out= routes at n=1/1K/100K, per-chunk, the 48-op A/B against the previous commit, the official copycast/pathology sections), the ranked next levers, and the gates to run. benchmark/nditer/probes/ holds the probes the document cites, as .NET 10 file-based apps with a RELATIVE #:project so they run from any checkout (verified: all three build and run from the repo root): fixed_cost_probe.cs construction + dispose vs the allocator floor; per-chunk driver on narrow rows; 1M/10M through the iterator vs raw vs np.*; the production out= routes with managed bytes per call; the glue components ab_ops_probe.cs 48 NDIter-routed production ops at 1K/100K as id<TAB>ns rows, with the git-worktree A/B recipe in its header angles_probe.cs the candidate next levers as experiments (narrow rows EXLOOP vs BUFFERED, where= run lengths, fancy index vs take/put, the fresh-NDArray floor, page-offset stagger INTERLEAVED) numpy_twins.py the NumPy 2.4.2 side (fixed | ab | angles) and a join that prints before / after / gain / NumPy / NPY/NS per row Two findings the document records that were NOT in the previous commit: - The 4K-aliasing "win" is rejected. A one-shot probe read the same V256 add on the same 1M pooled buffers (all at page offset 64 mod 4096, NumPy's too) at 0.460 ms natural vs 0.355 ms staggered. Alternated 5x best-of-15 on an idle host: 1M natural 0.388-0.411 vs staggered 0.383-0.454, 4M 3.08-3.63 vs 2.89-3.31 -- no effect outside the noise band; the first measurement ran cold and first. angles_probe (5) and its twin are written in the interleaved form, and the document carries it as measuring trap 9: any placement, alignment or prefetch claim needs the interleaved repeated form. - The ranked next levers, each with its measured experiment: fancy index a[idx] / a[idx]=v (252 / 294 us at 100K) vs the take/put kernels (99.6 / 82.7 us; NumPy 168 / 193) = 2.5-3.5x by routing the 1-D int-index-on-axis-0 indexer to them; a 2-D kernel contract for narrow rows (BUFFERED windows measured slower than per-row EXLOOP: w=4 4.45 vs 3.95 ms, NumPy 2.99); a SIMD run scan for the where= driver (64-blocks 38.5 vs 26.9 us, half-run 34.4 vs 21.9, alternating already 247 vs 288); the fresh-NDArray floor (np.empty(1000) 216 vs 183 ns, Shape is 5.5 ns of it); kernel-level gaps for a kernel pass (sum(f32,dtype=f64) 75.6 vs 37.0 us, amin axis 19.7 vs 14.7, sqrt(int32) 110 vs 78); ~80 ns of iterator-internal trims; the np.nditer per-element view. benchmark/nditer/README.md gains a "Probes" section; .claude/CLAUDE.md's NDIter paragraph points at the document.
…5x NumPy
The per-chunk ForEach route drives a (rows, w) view with a contiguous inner axis
one row at a time under EXTERNAL_LOOP, paying the odometer advance AND the
per-chunk kernel's own SIMD-viability prologue on EVERY row — so a narrow w sat at
~0.82x NumPy (np.positive) while a hand-written 2-D loop (prologue once, inner
SIMD run, per-row pointer bump) measured 1.7-2.1x. The whole gap was per-row
overhead, not memory traffic; buffering the strided rows into a contiguous window
was measured SLOWER (the inner runs are already SIMD-able in place). This is
docs/NDITER_PERF_DISCOVERY.md §7 angle 2.
ND2DElementwiseKernel (DirectILKernelGenerator.InnerLoop2D.cs) loops the outer
axis itself with per-operand outer byte strides, called once per coalesced 2-D
block — neither the odometer nor the prologue runs per row. It reuses the SAME
scalar/vector emit bodies as the per-chunk kernel, so results are byte-identical
(elementwise ops carry no cross-element state). Dispatched by
NDIterRef.{Is2DElementwiseShape,TryExecute2DElementwise}, wired into the three
packed-key ExecuteElementWise* entry points so it serves both out= and allocating
unary/binary routes. Gate: unbuffered, no where= mask, EXTERNAL_LOOP, inner axis
element-contiguous for every operand, all operands the same SIMD-capable dtype.
NumSharp's NDIter does not coalesce outer axes (a 3-D x[:,:,:w] stays NDim=3), so
the gate also flattens mutually contiguous outer axes
(stride[d]==stride[d+1]*shape[d+1], true for any trailing-narrow N-D slice) into
one (outerCount, outerStride) and reuses the same kernel — covering N-D
trailing-narrow AND routing it off a pre-existing malformed-output crash in the
ForEach 3-D odometer path.
positive had no vector body (CanUseUnarySimd(Positive) was false), so it was
scalar even contiguously and could not engage — an identity vector body was added
(EmitUnaryVectorOperation Positive branch + the gate), which also gives contiguous
positive a SIMD copy (1.59x NumPy).
Measured (NPY/NS, 2M f64): 2-D positive 1.95-2.18x, sqrt 2.11-2.52x, add
1.82-2.29x; 3-D trailing-narrow 1.37-2.26x — every cell a win.
Gates: FuzzMatrix 98/98 (incl. out_where across every strided layout), main suite
14446/0, + a 700-case 2-D and 123-case N-D self-consistency check (strided result
bit-identical to the contiguous-copy result across ops x dtypes x widths x
layouts). Both TFMs build.
Still on the per-chunk route (correct, unaccelerated): inner-broadcast 2-D
(add(A, col), stride-0 inner) and non-flattenable outer (doubly-strided
x[::2,:,:w]).
What/how write-up companion to af25a74: the problem (per-row odometer + prologue on a contiguous-inner/strided-outer view), the investigation (baseline 0.82x, the hand-2D floor proving the gap is overhead not bandwidth, buffering rejected), every angle reviewed, the fix (ND2DElementwiseKernel + dispatch gate + N-D outer-axis flatten + the positive SIMD identity body), the results (1.4-2.5x across 2-D and N-D, positive contiguous 1.59x), the gates (FuzzMatrix 98/98, main 14446/0, 700+123 self-consistency), the pre-existing np.empty(Shape) crash found along the way, and reproduction.
Regression fix for 3d4bc47. That note told the model to write a large or quote-dense probe to a file and redirect it in, but gave bare relative examples (probe.cs / probe.py). A bare relative name resolves against the repo cwd, so probe files landed in the repository tree instead of the session scratchpad. Fix: the note now says to write the probe file in the session scratchpad directory (path from the system prompt; never the repo cwd, never /tmp; a bare probe.cs lands in the repo tree) and the examples use <scratchpad>/probe.cs and <scratchpad>/probe.py. Timing/benchmark scripts are unchanged and still defer to the benchmark skill. Also corrects the stated cause: the Bash tool wraps every command in a single-quoted eval and re-escapes each single quote, so a quote-dense heredoc stops balancing at scale; the earlier transport-corruption wording was a guess.
…rows, inner-broadcast modes, scalar-only block, per-block odometer; NDIter coalesces axes like NumPy Review of docs/NDITER_2D_BLOCK_KERNEL.md (commit af25a74) with a wider probe than the first report used. The report measured one dtype (f64), one size regime (2M, DRAM-bound), widths of at least one full vector and ops that have a vector body; a probe over widths 1-64 at 100K/1M/2M, f32/u8/i32, ops without a vector body, the broadcast shapes it listed as "left" and 3-D views found three regimes where the kernel's own shapes were LOSING to NumPy, plus a general iterator defect. All measured pinned to one P-core (NS_PROBE_AFFINITY), old kernel built in a detached worktree, runs interleaved old/new/old/new, min of two rounds per side, NumPy pinned identically (NPY/NS, higher = NumSharp faster). Findings (before): * sub-vector rows: np.sqrt(m[:, :2]) / (m[:, :3]) ran the SCALAR tail per row while NumPy finishes rows with a masked vector -> 0.53x / 0.41x NumPy at 100K, 0.60x / 0.52x at 2M * inner-broadcast: add(A, col) 1.15x (vs 3.3x for add(A, row)), multiply(view, 2.0) 0.48x, add(view, col) 1.63x — the gate demanded a contiguous inner axis for EVERY operand, so a broadcast column / 0-d scalar fell to the per-row route at ~7 ns/row * ops without a vector body: np.exp on 4-wide rows 0.84x (+43% over contiguous) — the gate demanded a vector body * iterator defect: NDIter only ran CoalesceAxes when every operand was contiguous AND nothing was broadcast (that routine assumes the ascending pre-coalesce layout its own sort makes), so a C-contiguous (250000, 4) array times a 0-d scalar iterated 2-D, one kernel call per 4-element row: 1.7 ms for 1M elements. NumPy's npyiter_coalesce_axes runs unconditionally; the oracle already listed the consequence as the K1 "coalesces fewer dimensions" known bug. Changes: * NDIterCoalescing.CoalesceAxesIterationOrder — NumPy's coalesce for the iteration-order layout (innermost axis last): merges every adjacent (outer, inner) pair that ALL operands walk as one axis (stride[d] == stride[d+1]*shape[d+1], or a size-1 stride-0 axis), so the element-visit sequence is unchanged and only the odometer shortens. Called on the non-all-contiguous construction branch in place of RemoveUnitAxes (which it subsumes). multiply(A, 2.0, out) on a contiguous (250000, 4): 1.7 ms -> 251 us (one 1-D chunk); a[:, ::2] is ONE strided chunk exactly as NumPy's external_loop hands it out; a 3-D x[:, :, :w] reaches the kernel as (rows, w) with no special flatten. Probed against 2.4.2 on the three oracle K1 layouts x C/F/A/K orders: chunk lengths and value streams identical except the pre-existing 'A'-order transposed-3-D case, so the K1 excuse is narrowed to that one layout and strided_2d_cols / negstride_2d_offset are gated bit-exactly. * ND2DElementwiseKernel contract gains innerByteStrides; the kernel dispatches ONCE per call on the inner-mode pattern — unary {C, S}, binary {CC, SC, CS} — each a full 2-D loop with the S (broadcast-along-the-row) operand loaded and Vector.Create'd once per row. The gate accepts inner stride 0 on inputs (output must be element-contiguous). * Masked sub-vector rows and tails: on a 256-bit AVX2 host with 32/64-bit lanes the row remainder innerCount % lanes — and the whole row when innerCount < lanes — is one MaskLoad per C input -> the same vector body -> one MaskStore (vmaskmov, NumPy's npyv_load_tillz / store_till). The mask is built once per call (GreaterThan(Create(tail), {0,1,..})). Masked-off lanes are never touched in memory; integer masked-off lanes are filled with 1 after the load so a software Vector.Divide cannot throw on a 0 lane. Other hosts and 1/2-byte lanes keep the scalar tail. * Row-shape dispatch once per call: generic (unroll/remainder/tail), one-vector rows (innerCount == lanes: one full vector per row and NO inner loop — the generic shape paid three loop compares and an index update per row for the same work, 2.03 vs 1.59 ns/row on 1M f64 w=4), or masked rows (innerCount < lanes). * Scalar-only 2-D block for a null vector body or a non-SIMD dtype set (constant-stride addressing when every operand's inner stride is its own element size, runtime strides otherwise) — exp/log/power/mod/mixed-dtype/Half/Decimal/Complex no longer pay the per-row route. * Leading axes that do not fold into the block (x[::2, :, :w]) are walked in TryExecute2DElementwise by a per-BLOCK odometer, never per row. Results (old -> new, NPY/NS): sqrt w=2/3: 0.53/0.41 -> 1.58/1.84 (100K), 0.55/0.42 -> 1.65/1.90 (1M), 0.60/0.52 -> 1.78/2.24 (2M) positive w=2/3 1M: 3.3/2.4 -> 3.9/3.1; add w=2/3 1M: 2.5/2.1 -> 3.1/2.8 w=4 (one-vector rows): 2M positive 1.91 -> 2.33, sqrt 1.99 -> 2.38; 1M add 2.2 -> 2.6 exp w=4 1M: 0.84 -> 1.22 (narrow view now costs what the contiguous one does); mod 1.87 -> 2.41 add(A, col) c=4: 1.15 -> 6.5; c=16 1.35 -> 2.6. multiply(view, 2.0): 0.48 -> 1.97. add(view, col): 1.63 -> 6.0. view * 2.0 (allocating): 1.10 -> 4.8. add(A, row) unchanged 3.3. 3-D x[::2, :, :4]: positive 0.70 -> 2.12, add 1.11 -> 2.45; x[::2, :, :16] 1.02/1.24 -> 1.49/1.79 Every w >= 4 cell of the first report is unchanged or up. Measurement traps recorded (docs/NDITER_PERF_DISCOVERY.md §3 trap 11): on this hybrid P/E-core host an unpinned benchmark thread reads 2-3x slower uniformly and swings run to run (1M positive w=4: 800-2600 us unpinned vs 411 us pinned — the first report's absolute numbers did not reproduce); narrow_probe.cs and numpy_twins.py honour NS_PROBE_AFFINITY. Cache-resident strided-stream cells at 100K move +-20% with the same binary from placement alone. Gates: 16,831-check self-consistency probe (every mode x widths 1-100 x 10 dtypes x reversed / strided-out / aliased-out / mixed-dtype / 3-D flat, ::2, ::3, ::-1, two-axis-strided / 4-D / bool / Complex, bytes vs the same op on contiguous copies) 0 mismatches over 124 compiled 2-D kernels; FuzzMatrix 98/98 (K1 narrowed); main suite 14446 passed / 0 failed / 11 skipped; iterator subset 907 passed + the same five [OpenBugs] as before; net8.0 + net10.0 build. One test updated: AxisStride_2D_NonContig_NoMultiIndex_FortranOrder pinned the old non-coalescing (NDim == 2 for a[:, ::2] under K order); NumPy reports ndim 1 and a single chunk there, so it now asserts that. Still open: copy-class ops at w=2/3 gain only 1.1-1.3x from the masked path (a half-vector store for w == lanes/2 would add ~1.3x — a per-width specialization, not done); where= masked and buffered-cast routes are untouched; a strided-inner row that does not coalesce stays on the per-chunk gather path; the K1 'A'-order axis ordering on a transposed 3-D operand remains. Files: Backends/Iterators/NDIterCoalescing.cs (+CoalesceAxesIterationOrder), NDIter.cs (call site), NDIter.Execution.Custom.cs (gate + dispatcher + per-block odometer), Backends/Kernels/Direct/DirectILKernelGenerator.InnerLoop2D.cs (the second-generation kernel), test/NumSharp.Tests.Oracle/Fuzz/MisalignedRegistry.cs (K1 narrowed), test/NumSharp.Tests/Backends/Iterators/NDIterAxisStrideArrayTests.cs, benchmark/nditer/probes/ narrow_probe.cs (new) + numpy_twins.py (narrow twin, affinity pin), docs/NDITER_2D_BLOCK_KERNEL.md (§11), docs/NDITER_PERF_DISCOVERY.md (§3 trap 11, §7), .claude/CLAUDE.md.
…here= SIMD run scan, where= in the 2-D block kernel; coalescer reach verified Remeasures and closes the four Tier 1 levers of docs/NDITER_PERF_DISCOVERY.md §7 (now §6.6). Every number below is pinned to one P-core (NS_PROBE_AFFINITY=0x4), min-of-rounds, against commit ccadeef built in a detached worktree and NumPy 2.4.2 on the same core (NPY/NS). 1. The fancy-index operator runs the take/put kernels. a[idx], a[idx] = v, m[ridx], m[ridx] = row — ONE integer index array over a C-contiguous source of any rank — go through Selection/FancyIndexKernels.cs instead of the MapIter-style route (an offset delegate per element, a materialised offset array, a second pass): * DirectILKernelGenerator.GatherFlat.cs: lean flat gather/scatter kernels for primitive-width slabs — compile-time element width (one shift, no runtime-size multiplies), running index/destination/values cursors, NumPy's check_and_adjust_index as one add + one UNSIGNED compare, RAISE-only. NumPy's mapiter_trivial a[idx64] is 57 µs at 100K; the general take kernel sat at 75 µs and could not beat it, the flat one does (45 µs). * The general take/put kernels gained idx32 variants: int32 index arrays are read IN PLACE (np.array(new int[]{…}) is int32); np.take/np.put use them too, shedding the widening copy that was 27 % of np.take(a, idx32). * Whole trailing sub-arrays per index (m[ridx]) use the general kernels with a cpblk slab; m[ridx] = row scatters one sub-array through the wrapping values cursor with no materialised broadcast. * The SETTER pre-validates every index with a Vector<T> bounds scan before its first store, exactly as NumPy's mapiter_trivial_set does, so a bad index leaves the array untouched. Verbatim IndexError, both value-broadcast ValueError texts, last-write-wins on duplicate indices, view offsets on source and index array, all 15 dtypes are pinned by test/NumSharp.Tests/Selection/Indexing.KernelRoute.Tests.cs. 100K f64: a[idx32] 0.81→4.97×, a[idx64] 0.22→1.26×, a[idx64]=v 0.37→1.13×, m[ridx] 1.40→2.68×, m[ridx]=row 2.10→6.29× (10M: 0.33→1.27×), np.take(a, idx32) 3.6→9.5×. 2. The where= masked driver scans mask runs with SIMD (NDIter.Execution.cs, InvokeInner). One movemask per 32-byte block; the runs inside a block are walked with two trailing-zero counts, a run reaching the block end continues through a vector skip. A first version that probed 32 bytes per run boundary regressed run-length-1 masks (237→353 µs); the per-block bit walk keeps both regimes. 100K f64 add: all-false 27.7→1.5 µs (3.5× NumPy), half 34→8.2 (2.8×), 64-runs 43→16.9 (1.4×), 1024-runs 37.8→10.7 (2.5×), all-true 55→26 (1.9×), run-length-1 unchanged (1.1×). 3. where= admitted into the 2-D block kernel (DirectILKernelGenerator.InnerLoop2D.Masked.cs; NDIterRef.Is2DMaskedElementwiseShape / TryExecute2DMasked / Run2DMaskedBlocks, wired into BOTH ExecuteElementWise overloads — the ufunc routes arrive with a packed InnerLoopKernelKey, which a gate on the string-key overload alone never sees). The mask rides as the trailing operand; a byte-per-element mask becomes each vector group's lane mask (Avx2 pmovzx + GreaterThan(·,0)) feeding the same MaskLoad → body → MaskStore the unmasked kernel uses for its tails (integer masked-off lanes 1-filled); a (rows,1) mask gates whole rows; ops without a vector body or non-SIMD dtypes get a masked scalar 2-D block; the tail word is assembled from exactly `tail` bytes so the mask array is never over-read. C/S patterns {C,S}/{CC,SC,CS} as the unmasked kernel. 1M f64 (rows, w) views: w=3 add byte-mask 0.66→1.67×, row mask 1.82→4.11×, column mask 0.99→3.50×, sqrt 0.38→1.29×; w=4 add 0.78→2.17×, column mask 2.62→10.5×, sqrt 1.26→6.24×; w=16 add 2.30→3.60×; unmasked rows unchanged. Gate: an 8,358-check self-consistency probe (11 dtypes × 14 widths × 6 mask kinds × 6 ops × 2 layouts + 3-D / stepped 3-D), every strided result equal to the same call on contiguous copies AND to an independent np.where composition, 0 mismatches. 4. The coalescer's reach on copy, cast and reduce — verified, nothing to do. benchmark/layout + benchmark/cast benches, pinned, pre-coalescer fb6e34c vs ccadeef: reduce 0.997× (864 cells), copy/identity 1.031× (464), elementwise 1.068× (672; strided 1.11×), cast 0.992× (832). The per-cell 0.45–2.2× swings both ways are the documented allocator-regime noise (C-layout cast cells swing too, and a 1-D contiguous cast cannot be routed differently). Two measurement traps recorded (discovery doc §3 traps 12–13): `/` on an integer NDArray is NumPy TRUE division, so `(ar / 64 % 2) == 0` built a SPARSE mask (one true per 128), not 64-runs — the first where= numbers compared different masks on the two sides (angles_probe.cs fixed to np.floor_divide); and never A/B the live tree — a chain of `dotnet run` benches compiles the tree as it is when each run starts, so editing mid-chain hands later runs a half-edited tree. Probe: benchmark/nditer/probes/fancy_where_probe.cs + numpy_twins.py fancy_where (sections A/B/C). Gates: main suite 14476/0 (CI filter), FuzzMatrix 98/98 (out_where.jsonl covers every ufunc out=/where= layout), Indexing.KernelRoute.Tests 30/30, net8.0 build.
…asurement — probe section D NumSharp indexes with int64/long everywhere; a narrower index path is admissible only if it measurably increases performance. Measured the idx32 take/put/flat kernel variants (index array read in place at 4 bytes, sign-extended on load, all index arithmetic int64) against the int64-only alternative — widen the int32 index array to a contiguous int64 temp, run the int64 kernel — and against a native int64 index, pinned to one P-core, min-of-rounds: widen / in-place: 1.7–2.5x at 1K (the temp's allocation floor), 1.1–1.8x at 100K (its convert + write), 1.2x at 10M (its extra 12 bytes per index of memory traffic); native int64 / in-place: 0.97–1.13x (an int32 index costs the same as an int64 one). So the variants stay, as a load-width-only specialization that removes a copy and never becomes the type anything computes in; drop them if the widening ever becomes free. Recorded in docs/NDITER_PERF_DISCOVERY.md §6.6 (table) and .claude/CLAUDE.md; reproducible as section D of benchmark/nditer/probes/fancy_where_probe.cs (C# only — an internal comparison, no NumPy twin).
…maining gaps, ranked plan L1–L8; neighbours probe docs/NDITER_PERF_CONTINUATION.md is the next session's entry point for the NDIter / fancy-index performance line: where the tree stands (09d1fc5 + bce0a2f, gates), the measurement protocol (pinned, detached-worktree A/B, the probe/twin table), what is still slow MEASURED rather than remembered, and a ranked plan with a concrete design per lever. Measured neighbours of the Tier 1 levers (benchmark/nditer/probes/neighbours_probe.cs + numpy_twins.py neighbours, pinned, NumPy 2.4.2, NPY/NS): fancy shapes still on the delegate route — two index arrays m[ri,ci] 0.09x (100K), slice + index array m[:, rk] 0.05x, m[rk, :2] 0.26x, F-order source rows mT[rk] 0.18x, strided index view 0.21x, their setters 0.11–0.15x; boolean masks / compress / take(axis=1) / put-into-view are 2–17x ahead (no action). where= neighbours — only the cast-out masked narrow rows lose (0.33x; the buffered route is excluded from the block kernels); copyto(where=) 0.86x; comparison where=, np.where, no-out where= are 2–4x ahead. the Tier 2 claims — buffered-cast unary routes are at parity or ahead (sqrt(i32, out=f64) 1.03x; the 0.71x was unpinned); sum(f32, dtype=f64) is 1.0x vs NumPy but 24x slower than our own sum(f32) (a converting scalar loop); trailing-axis narrow-row reductions are fine (sum 1.1–1.8x, max 1.5–3.9x, mean 0.95–1.0x) and the real gap is the LEADING axis with few rows (sum(axis=0) on (3, N) 0.52x, (4, N) 0.58x); tiny-op fixed cost is 2–23x AHEAD of NumPy. The plan: L1 one index array + slices → take/put-axis kernels (put needs an axis kernel); L2 an offsets kernel for two+ index arrays feeding the flat gather (any source layout); L3 a strided-slab gather (non-contiguous source rows; also what np.take should use instead of ascontiguousarray); L4 copy a strided index view then the kernel (trivial); L5 the few-row leading-axis reduction kernel (floor probe first; flat ~0.6 ns/element regardless of cache residency says overhead, not bandwidth); L6 cast-out where= as two passes (masked block kernel into a loop-dtype temp + masked cast copyto); L7 a fused widening flat sum (parity risk: a buffered NumPy reduction is pairwise per 8192-element window, sequential across windows — gate it in the reduce oracle tier first); L8 the copyto(where=) run scan. Skip list with numbers. Traps 1–9 (packed-key overload, `/` is true division, never A/B the live tree, pin, NumPy's 1M allocating cells, runfile cache, np.take copies a non-contiguous source, Storage.Address vs Shape.offset, bufferedPromoting is a cast + wide kernel). Gates and a file map. Also: the discovery doc's §7 points at the continuation document; CLAUDE.md's NDIter section names it as the next session's starting point.
… into NDW012, [NDBorrowed]
Why
- NDW012 is per-method and treats "stored into a field/property" as a clean escape, so
ownership silently evaporated at every field boundary: a class could hold NDArrays until
the finalizer and nothing ever asked it to be IDisposable or to dispose them. Eli's rule:
a type that stores NDArrays in any way must be disposable and dispose them, and a type
storing such a disposable must dispose it too (contagious disposing responsibility).
What — a third analyzer in tools/NumSharp.Build.Analyzer (same DLL, same package payload)
- NDArrayHolderAnalyzer.cs over the shared OwnershipModel.cs (memoized, concurrent-safe):
* NDW016 (warning, one per type): a class/struct declares instance fields / auto-properties
(positional record properties included; a computed property is a view) whose type HOLDS
NDArrays — NDArray(<T>)/subclass, arrays of any rank, tuples with a holding component,
generics instantiated over a holder (List/Dictionary/Lazy/Task/a consumer's Box<NDArray>,
and types nested in such containers), INDArrayCarrier structs, T : NDArray, and ANOTHER
type that stores them (own members or base — ownership is contagious) — yet implements
neither IDisposable nor IAsyncDisposable (a ref struct: no public void Dispose()).
* NDW017 (warning, one per member): the type is disposable but the holder is not disposed
on any path from Dispose/Dispose(bool)/DisposeAsync/DisposeAsyncCore. Reachability runs
over same-type callees (methods, property accessors, delegates over methods; local
functions are in-tree). Disposal is recognised when a Dispose/DisposeAsync/Close/close
receiver ROOTS at the member (through conversions, ?., tuple/field/property/indexer/
element chains, call results, and derived locals: declarators, assignments, foreach loop
variables incl. deconstruction, deconstruction — to a fixed point), the member is a
`using` resource, it is handed as an argument to any method/ctor (type-gated: the
argument itself must hold NDArrays and not be a call result, so Log(x.size) and
Log(_a.ToString()) are not disposals), or a call on it carries a disposing lambda
(_list.ForEach(x => x.Dispose())). Not disposal: `= null`, Clear(), a finalizer (the
backstop, deliberately not a root), an unreachable helper. Message hints name an
inherited base Dispose (override Dispose(bool)) or a non-disposable inner type (make it
disposable first).
* Exempt: static classes, interfaces, INDArrayCarrier types (transient result packaging the
scope yields through), [NDBorrowed] types. Never owners: static members, delegates,
comparers/IEquatable/IComparable/IObservable/IObserver/IProgress, WeakReference<T>,
unconstrained type parameters, object/IDisposable, any type from an assembly that cannot
name NDArray (BCL answered without a member scan). Metadata types are consulted (visible
fields/properties/indexers, so np.NDIterator and NpzFile count), never reported.
- NDArrayLeakAnalyzer (NDW012) contagion: an NDArray-owning disposable (a consumer holder
class, np.NDIterator, NpzFile) or a rank-1 array of them is an owned value — a dropped/dead/
discarded/factory instance warns; Dispose/DisposeAsync/Close/close reclaim; return/store/
hand-off escape; a foreach over a PRODUCED NDArray or owning disposable is a leak (C#
disposes the enumerator, never the enumerable — foreach (var x in np.nditer(a)) leaves the
iterator open) while a produced NDArray[]/tuple/carrier stays conservatively consumed. The
message names the kind ('Batch' instance / 'NDIterator' instance / 'Batch' array).
- [NDBorrowed] (src/NumSharp.Core/Backends/NDBorrowedAttribute.cs; Field|Property|Class|
Struct; runtime-inert like [NDScopedCovered]): "this references an array owned elsewhere"
— excludes a member, or exempts a whole type and removes it from the contagious set. The
name contains neither NDW013 scan token (NDScopedAttribute/NDScopedAsyncAttribute).
- KnownTypes: BorrowedAttr, IDisposable, IAsyncDisposable.
Core triage (fresh -t:Rebuild: 18 NDW016 types + 9 NDW017 members + 9 new NDW012 sites →
exactly 1 NDW012, the documented indexer false positive, 0 NDW016, 0 NDW017)
- [NDBorrowed] on genuine borrowers: FlatIterator, MemoryView, NDRefIter<T>/NDChunkIter<T>,
NDEnumerate(<T>), NDFlatIterator, NDExpr ArrayNode/NDExprBindContext, NDArrayFlags, the
debugger proxy, _Unsafe/_Pinning, r_'s Operand, SplitContext, IndexOp/PreparedIndex,
NpzFile.BagObj, NDIterRef._operands/_writebackOriginals (an iterator never owns its
operands; the COPY_IF_OVERLAP temp is resolved by ResolveWritebacks — this also stops the
five Borrow() handle drops from reading as leaks), NDIterator.Current + its foreach
Enumerator, NDArray.TrackingScope (the scope owns the array, not vice versa),
Broadcast._ops, NpzFile._cache (an array is the reader's the moment it is handed out;
disposing the cache on Close would break `using var z = np.load_npz(p); var a = z["a"];`).
- REAL ownership gaps the pass found and fixed:
* poly1d owned its coefficient array (the ctor yields it into the field via
scope.Returns) yet was not disposable → poly1d : IDisposable; the copy-constructor now
COPIES (+ NDScope.Detach, a field egress) instead of sharing one array between two
owners that would double-dispose; operator +/-/*// (poly1d, NDArray) normalization
temps are `using`; polyval(poly1d, poly1d) disposes its Horner product and superseded
accumulator. (Operators cannot be [NDScoped]: a poly1d return is an NDW003 carrier.)
trim_zeros returns this[Slice[]] = always a new view object, so _coeffs is never the
caller's instance and Dispose stays R2-safe.
* Broadcast.Dispose() (what foreach calls at loop end) was a no-op while the object owned
the broadcast_to views it built lazily → now releases them and nulls _views/_iters,
which rebuild on the next access (still re-enumerable; operands untouched; each view
holds its own ARC ref).
* IndexCollector (public struct, no call sites) stranded its outgrown buffer on growth and
on the trim path → IDisposable; ToResult hands the storage out on a perfect fit.
* np.isreal/np.iscomplex were unscoped entries handing an np.imag view + a Scalar(0) temp
to np.equal/np.not_equal → [NDScoped] (NDScope.Returns re-attaches to the parent scope,
so results are unaffected).
Build/config
- NumSharp.Core.csproj: NDW016;NDW017 join NDW012 in WarningsNotAsErrors (nudges, never
build-breakers); src/NumSharp.Core/.editorconfig and the fixtures .editorconfig carry the
three severities explicitly; comments updated.
- tools/verify_build_package.sh: step 16b — a real PackageReference consumer draws NDW016 on
a storing type and NDW017 on a forgetful disposable, and nothing on a disposing type or an
[NDBorrowed] type/member (18/18 steps green end-to-end).
Tests — 216 in-process + 4 AnalyzerBuild, green on net8.0 and net10.0
- Fixtures (compile standalone in the harness): HolderTypeScenarios.cs (NDW016 tags on the
TYPE lines — 25 warn types incl. records, structs, ref structs, abstract, generics,
carriers, Lazy/Task, np.NDIterator, NpzFile, cyclic pairs; 25+ clean exemptions incl.
NumSharp's own [NDBorrowed] FlatIterator seen through metadata), DisposePathScenarios.cs
(NDW017 tags on MEMBER lines — 14 forgotten shapes; 40+ clean dispose spellings),
ContagionScenarios.cs (9 NDW012 drops incl. np.nditer and foreach; reclaim/escape/exempt
twins). EscapeScenarios.Holder now carries its NDW016 tag.
- HolderTypeTests / DisposePathTests / ContagionTests: exact-match gates + content facts +
message pins (member list "and N more", struct/ref-struct hints, contract names, the
inherited-Dispose and make-inner-disposable hints, the instance/array descriptions).
- OwnershipMetamorphicTests (19 pairs): IDisposable turns 016 into 017, the dispose call
clears it, IAsyncDisposable, [NDBorrowed] member/type, static, computed, carrier, ref-struct
pattern, a chain resolving one level per fix, contagion stopping at [NDBorrowed], `using` a
dropped holder, non-disposable holder moves the verdict to the type, close(), foreach until
`using`, helper reachability, overriding an inherited Dispose, foreach-dispose of a list.
- OwnershipPropertyFuzzTests: 120 seeded types (1–6 members from a 14-template grammar,
disposable or not, random holder subset disposed through 3 Dispose shapes) — NDW016 ==
non-disposable holder types, NDW017 == undisposed holders, NDW012 == 0.
- OwnershipRobustnessTests (19): cycles with/without arrays, self-reference, 25-deep chains,
generic definition vs Box<NDArray>/Box<int>, alias, nullable, nested, partial, 300 members,
200 types, static/interface/enum/delegate, no-NumSharp no-op, malformed, unresolved member
types, arrays of holders, Cell<NDArray>, other-instance helper.
- AnalyzerContractTests: NDW016/NDW017 are Warnings, enabled, anchored; disjoint ids across
the three analyzers. HarnessSelfTests tag floors + code presence for the new fixtures.
- Core regression subset (poly1d, Broadcast, nditer, isreal/iscomplex, NpzFile, ndenumerate,
flatiter, memoryview, ScopeAudit, StrongName): 1384 passed, 0 failed.
Two guardrails the fixtures forced: the hand-off rule MUST be type-gated (without it,
Console.WriteLine(x.size) on a loop element silently counted as disposing the list), and
HoldsNDArrays memoizes only root answers or nested answers not cut by a cycle (a cycle-cut
nested answer memoized as false turned a real holder clean).
Docs: docs/website-src/docs/numsharp-build-compiler.md (NDW016/NDW017 sections, contagion
note under NDW012, code table rows, analyzer coverage line), docs/LEAK_ANALYZER.md (§10 the
ownership pass + §11 verification), COVERAGE_PLAN.md §11, ARCHITECTURE.md, CI workflow
comment, RELEASE_0.70.0.md (breaking: poly1d IDisposable + copy semantics; Broadcast.Dispose;
IndexCollector).
Relocate design notes and planning docs that are no longer the current source of truth into docs/stale-docs/, so the live docs/ tree lists only current references. Pure file moves (git renames, content unchanged). Moved (root/docs design notes): - DISPOSAL-GUIDELINES.md - docs/FFT_PARITY.md - docs/FLOAT16_DESIGN.md - docs/GEMM_PARITY.md - docs/LEAK_ANALYZER.md - docs/NDITER_2D_BLOCK_KERNEL.md - docs/NDITER_PERF_CONTINUATION.md - docs/NDITER_PERF_DISCOVERY.md - docs/OPENBLAS_DELIVERY_DESIGN.md - docs/UNIQUE_DESIGN.md - docs/UNMANAGED_STORAGE_UNION_DESIGN.md Moved (docs/plans): - AUTOCOVERAGE_TESTFRAMEWORK.md - DESIGN_CHALLENGE_10_FUNCTIONS.md - EXCEPTION_REDESIGN.md - UNIFIED_ITERATOR_DESIGN.md - advanced-index-axis-placement.md - advanced-index-combinatorial-handover.md - fft-integration-plan.md - gh-issue-npy-npz-rewrite.md - indexing-numpy-parity-refactor.md - interop-docs-spec.md - numpy-1x-deprecation-audit.md - numpy-1x-deprecation-findings.md - offset-model-rewrite.md Note: several of the moved files are still referenced by path in .claude/CLAUDE.md (FLOAT16_DESIGN, GEMM_PARITY, UNIQUE_DESIGN, NDITER_PERF_DISCOVERY/CONTINUATION, NDITER_2D_BLOCK_KERNEL, OPENBLAS_DELIVERY_DESIGN, FFT_PARITY, UNMANAGED_STORAGE_UNION_DESIGN). Those links now resolve under docs/stale-docs/ and can be updated or dropped in a follow-up.
… + oracle gate Closes the NaN byte-identity divergences in docs/bugs/nan-byte-identity-vs-numpy.md (now deleted) and closes the oracle blind spot that hid them. ## Root cause (one line, proven on x86-64 SSE) .NET double.NaN is the NEGATIVE-signed quiet NaN 0xfff8...; NumPy's NPY_NAN is the POSITIVE 0x7ff8.... On win-amd64 NumPy's complex ufuncs run through MSVC's UCRT C99 complex functions, which emit the positive NaN in their "produce a NaN" special paths. Both engines lower to the SAME deterministic SSE2 ops, so NaN PROPAGATION already matched - every divergence came only from NumSharp's NaN SEED constants (double.NaN), from inf-inf/0/0 arithmetic where NumPy uses the constant, or from BCL Complex.Exp delegations for non-finite inputs. Verified: inf-inf/0-div-0/inf*0 -> 0xfff8 on BOTH engines (kept as arithmetic); a positive-NaN operand propagates through +,-,*,/,sqrt unchanged; an explicit negate flips the sign - so a per-path fix (positive-NaN constant where NumPy canonicalises, arithmetic where it produces a signed result) makes NumSharp bit-identical to NumPy. ## The fix (NDComplexMath.cs) - per-path, NOT a blanket canonicalize - positive-NaN constant NAN (0x7ff8...) replaces the double.NaN seeds in HypotNonFinite, CasinhNonFinite, CacosNonFinite, Catanh, ExpSpecial. - sqrt/log/log1p: HypotNonFinite now PROPAGATES the present NaN's sign (IsNaN(x)?x:y) - so sqrt(2.5,-NaN).real is -NaN like NumPy, while clog's pre-abs'd operands keep giving +NaN. - reciprocal: rewritten as the exact CDOUBLE_reciprocal ufunc loop (1/d, -r/d, r/d, -1/d). NumPy's imaginary term is a DIVISION (-1.0)/d, not the old Smith -scl negate - a negate flipped a positive NaN d. - square: true vfmaddsub via Fma.MultiplySubtractScalar (real) + Fma.MultiplyAddScalar(im, re, re*im) (imag). The fused subtract keeps the subtrahend's NaN sign (the old -(im*im) negate flipped it); re*im in the addend slot matches NumPy's product-NaN on mixed-sign-NaN inputs. Finite results (incl. square(1e300+1e300i).real = -inf) are identical. - exp/exp2: Complex.Exp now only serves fully-finite inputs; ExpSpecial reproduces MSVC cexp (PROPAGATES an input NaN's sign; +NaN for an Inf input; +0 imaginary zero for a NaN y at -inf). - sinh/cosh: CANONICALISE every NaN slot to +NaN (MSVC csinh/ccosh do too, verified csinh(-NaN,y)=+NaN) - which is what makes the derived csin/ccos match after their -Re transform negate; also fixes the cos(+-0,NaN) zero-sign. - tanh: PROPAGATES the NaN sign (ctanh(x,-NaN)=-NaN); both-NaN follows y. Result: bit-exact over the FULL +-0/+-inf/+-NaN x +-0/+-inf/+-NaN grid (BOTH NaN signs) for sqrt log log2 log10 log1p exp exp2 expm1 square reciprocal sin cos tan sinh cosh tanh arcsin arccos arctan arcsinh arccosh arctanh conjugate negative positive. Only the pre-existing finite-interior <=1-ULP residuals remain (documented, excused). float16 sign/maximum and complex/float16 NaN preservation were already byte-exact. ## The oracle gate (was blind - BitDiff tokenized every NaN to "NaN") - BitDiff.Compare(..., nanBitExact) overload compares NaN by RAW bytes; new DiffHasSignFlip distinguishes a pure NaN-sign / signed-zero flip from a value change (WithinUlp reports NaN-vs-NaN and +-0 as "0 ULP", which would swallow it). - FuzzCorpusTests.CompareArray turns nanBitExact ON for ComplexNanContractOps (the complex-unary set) and HARD-FAILS a sign flip before any ULP/pathological excuse runs. Everything else keeps the tokenizing compare (a differing NaN payload is non-contractual elsewhere). - The corpus already carried NumPy's exact NaN bits; the gate immediately caught the -NaN-input exp/exp2 cases my first pass missed (now fixed). FuzzMatrix 98/98 on net8.0 + net10.0. ## float32 sum NaN sign - ACCEPTED as documented (policy decision) sum(float32 NaN-laced) is 0xffc00000 vs NumPy's 0x7fc00000: an order-dependent NaN-BIT difference (value is NaN either way). Matching NumPy's pairwise NaN tree would abandon the multi-accumulator SIMD reduction for a non-contractual sign bit - not worth it. The oracle correctly tokenizes float NaN, so it never false-fails. Gates: FuzzMatrix 98/98 (Unary+Specials complex-unary NaN now raw-byte compared); complex/math unit tests 819/0; Math/Fourier/Kernels/Statistics 3502/0.
…ng the unary NaN sweep)
Follow-up to the complex-unary NaN byte-identity fix. A corpus-wide raw-byte NaN
audit (103,229 cases run with BitDiff.Compare(nanBitExact:true)) surfaced 73
(op,dtype) combos where NumSharp's NaN bits differ from NumPy under the tokenizing
compare. Of those, exactly TWO were clean, deterministic, contractual UNARY complex
ops in the same class as the transcendental family - now fixed and gated:
- abs / absolute (complex128 -> float64): NumPy's npy_cabs canonicalises a NaN
component (no infinity) to the POSITIVE NaN 0x7ff8...; NumSharp's Complex.Abs
emitted .NET's negative 0xfff8. NDComplexMath.Abs now returns the positive NaN for
that case (finite path unchanged, inf path unchanged). Deterministic +NaN for every
input NaN sign, verified vs NumPy 2.4.2.
- sign (complex128): NumPy sign(z) is (+NaN,+NaN) for ANY NaN component or both-inf
(a CANONICAL positive NaN, not z/|z| which would keep each component's own sign).
The Complex branch of EmitSignCall (DirectILKernelGenerator.Unary.Math.cs) gained a
NaN-magnitude guard -> (+NaN,+NaN), and its both-inf branch (which emitted the
NEGATIVE double.NaN literal - a latent bug) now emits the positive NaN.
Gate: added sign/abs/absolute to ComplexNanContractOps, and the nanBitExact trigger
now keys on a complex128 OPERAND (not just a complex result) so abs/absolute - which
return float64 - are raw-byte NaN compared too. FuzzMatrix 98/98 (both frameworks);
the audit re-run drops from 73 to 71 combos (abs + sign resolved).
The remaining 71 are all NON-contractual or already-documented and are LEFT as-is
(the oracle correctly tokenizes their float NaN):
- float reductions / order-statistics (sum/prod/mean/median/std/var/ptp/percentile/
quantile/average/max/min + the nan* variants, f16/f32/f64): the surviving NaN's
sign is order/partition-dependent (SIMD multi-accumulator vs NumPy pairwise/
introselect) - the same non-contractual class as the accepted float32-sum case;
matching it would abandon the SIMD reduction for a meaningless sign bit.
- complex reductions/scans/binary (cumsum/sum/prod/mean/median/nanprod/add/multiply/
power/correlate/convolve, complex128): impl-defined complex NaN ordering/propagation
- already excused in MisalignedRegistry (F5/#12/reduction-NaN branches).
- log1p (f16/f32/f64): computed as Log(1+x); NumPy calls the CRT npy_log1p - a
documented non-portable accuracy/NaN defect.
- floor_divide/mod (float16): NDDivision has no Half path - known bug, documented.
- sort (f32/f64): NumPy's SIMD sort canonicalises NaN, NumSharp's radix keeps .NET's
negative NaN - the documented set-ops-era sort/unique NaN divergence.
Gates: FuzzMatrix 98/98; Sign/Absolute/Complex unit tests 940/0.
…umPy
"Do NumSharp's functions produce NumPy's NaN?" — a permanent, committed corpus tier
that answers it, replacing the one-off audit used to find the complex NaN-sign
divergences.
gen_nan_oracle.py is STANDALONE (like gen_npy_oracle.py / gen_decimal_oracle.cs) so
it owns its own case numbering and never renumbers the shared gen_oracle.py corpus.
It runs every UNARY op for which a NaN output is reachable over the FULL special-value
grid - finite / +-0 / +-inf / BOTH NaN signs (+NaN 0x7ff8... AND -NaN 0xfff8...) - as
complex128 (the 64-element re x im cross-product) plus float16/32/64 lines, recording
NumPy 2.4.2's exact output bytes -> nan.jsonl (120 cases).
The C# harness replays it through the existing CompareArray NaN-contract policy:
- complex128 unary ops (ComplexNanContractOps: sqrt/log/.../sign/abs) are compared
BIT-EXACT on the NaN sign; a pure sign / signed-zero flip HARD-FAILS via
DiffHasSignFlip before any ULP excuse (NumSharp reproduces NumPy's MSVC-UCRT
per-path sign). abs returns float64, so the trigger keys on a complex OPERAND.
- float16/32/64 stay tokenized: the tier still gates that a NaN is produced (right
VALUE) exactly where NumPy does and that the non-NaN outputs are byte-exact,
without false-failing the non-contractual float NaN sign.
FuzzMatrix 98 -> 99 tiers, green on net8.0 + net10.0 (only ~18 excused unary-ULP on
the non-portable float expm1/log1p finite outputs). Teeth verified end-to-end:
reverting HypotNonFinite's sign-propagation turns the tier RED with an explicit
"NaN-sign/signed-zero contract violation" on sqrt/log/log2/log10, then restores green.
Regenerate (needs numpy==2.4.2): python test/oracle/gen_nan_oracle.py
…lex expm1 NaN sign, host-gate the nan tier, refresh inventories PR #628's CI was red on two jobs; the last run (origin f5c3974) predates 5 local commits (complex-NaN sweep + the new `nan` oracle tier), which introduce further failures of their own. This greens all of it, verified on Windows/x64: NumSharp.Tests 14476/0 (net8+net10), Oracle 176/0 + FuzzMatrix 99/99 (net8+net10), Analyzer 216/0 (net8+net10). 1. macOS `test` job — HalfArithKernelTests.Arith_Specials_DefaultQNaN_Inf_SignedZero The float16 add/divide widen-compute-narrow kernel takes the HARDWARE FP result untouched on non-NaN operand lanes (DirectILKernelGenerator.Binary. Arith.Half.cs), so a FRESH qNaN from 0/0 and inf+-inf carries the CPU's default-NaN sign: x86/x64 emit the NEGATIVE default (float32 0xFFC00000 -> half 0xFE00, = the numpy 2.4.2 win-amd64 wheel this kernel is byte-pinned to), AArch64 emits the POSITIVE default (0x7E00). The Avx2 SIMD path is gated, so arm64 runs the scalar HalfArithBits fallback into the same hardware division. The test now asserts the arch-appropriate default (DefaultQNaNHalf) so the x86 byte-parity gate stays strict while it holds on the Apple-silicon runner. Only ARITHMETIC generates fresh NaNs — the min/max/ptp/compare/round Half kernels PROPAGATE input NaN bits and are arch-independent, so no other Half test needs this. 2. Oracle `nan` tier on Windows — complex expm1 NaN sign (real Core gap) The new nan tier (added in the unpushed 56a5571, never run in non-Windows CI) HARD-FAILED even on Windows: np.expm1 of a NaN-containing complex input emitted NumSharp's negative double.NaN (0xfff8...) where NumPy 2.4.2 emits the positive 0x7ff8... — because .NET's Math.Cos/Sin/Exp always return the negative double.NaN regardless of the input NaN sign, so the expm1 = expm1(x)cos(y) - 2sin^2(y/2) + i exp(x)sin(y) arithmetic flipped a +NaN input to a -NaN result. NumPy's nc_expm1 PROPAGATES the first input NaN's sign (real takes priority over imag) into both output components; NDComplexMath.Expm1 now guards this exactly like ExpSpecial (raw input NaN propagated), verified bit-for-bit vs 2.4.2 over the full +-NaN x {finite,+-0,+-inf,+-NaN} grid. This completes the complex-unary NaN-sign sweep (760bb5b/298e7c60) which had missed expm1. The inf-only slots carry no NaN and keep the hardware path, whose x86 sign already matches. 3. Oracle `nan` tier off Windows — host-pin it (RunHostLibmCorpus) The tier records win-amd64/MSVC-UCRT bytes and bit-compares BOTH the complex NaN SIGN and finite-input transcendental VALUES (its grid crosses `finite` with the specials, so sqrt/exp/log/... over finite operands appear). Off Windows those shift with the local libm, and on a non-x86 CPU a genuine 0/0.inf-inf slot's fresh default-NaN sign flips (DiffHasSignFlip would hard-fail) — a platform artifact, not a NumSharp-vs-NumPy defect. Swapped RunCorpus -> RunHostLibmCorpus so Windows stays the strict reference gate and Linux/macOS go Inconclusive, exactly like the sibling Unary/Precision/Fft tiers (matmul_parity pattern). The author's original RunCorpus was an oversight — the tier had never hit non-Windows CI. 4. docs `NumPy API + Tests & Oracle inventories` job — stale snapshots coverage/generated was stale (copysign/getbufsize/interp added, np.array ndmin default 1->0) so the `diff coverage/generated` step failed, which short-circuited before test/inventory/generated could be verified/uploaded. Regenerated both via coverage/generate_coverage.py and test/inventory/ generate_test_inventory.py. The coverage regen is BYTE-IDENTICAL (sha256) to the ubuntu CI artifact from the failing run, proving the generators are deterministic across Windows/ubuntu (explicit newline="\n", .as_posix(), sorted()); test/inventory carries no backslash path leaks and LF endings.
…omplex NaN-sign contract; refresh benchmark coverage Follow-up to 8e1f3cd. On the previous run Windows+Ubuntu went green and only test(macos-latest) failed — my Half fix let the suite run past HalfArithKernelTests and unmasked one more arm64 divergence; the docs job advanced past the (now-fixed) inventory diffs to a later stale-artifact step. Both closed here, verified in a clean worktree off origin/journey3 (both test projects build net8+net10; FuzzMatrix 0 fail). 1. NumSharp.Tests(macOS) — NewDtypesUnaryTests.Complex_Square_FmaContraction NDComplexMath.Square dispatches on System.Runtime.Intrinsics.X86.Fma.IsSupported: with x86 FMA it issues vfmaddsub so fma(re,re,-(im*im)) exposes im*im's rounding and square(1e-10+1e-10j).Real == -2.275e-37; WITHOUT x86 FMA (AArch64) it takes the non-fused fallback where re*re and im*im each round to the same 1e-20 and cancel exactly to 0. The test asserted the FMA residual unconditionally, so it failed on the Apple-silicon runner. Gate the residual assertion to Fma.IsSupported (mirroring the kernel's own dispatch, the numpy 2.4.2 win-amd64 reference), assert 0.0 on the fallback, and keep the imaginary/interior/overflow checks on every host. 2. Oracle(macOS) — the complex NaN-SIGN contract, gated to the x86 reference The unpushed complex-NaN commits (760bb5b/298e7c60) made CompareArray compare the complex-unary NaN SIGN by raw bytes (ComplexNanContractOps + DiffHasSignFlip) for EVERY tier, not just the new nan tier — specials(569)/unary_extra(494)/tail(54) all carry complex128 cases. That sign is authored against win-amd64/MSVC-UCRT and holds on the whole x86 family (Windows AND Linux, shared SSE) but NOT on AArch64, where a fresh NaN from a genuine 0/0.inf-inf inside the complex arithmetic carries the CPU's POSITIVE default sign (x86 emits NEGATIVE) — a DiffHasSignFlip hard-fail on a platform artifact. Those tiers' finite complex VALUES are proven cross-platform (macOS-green at 02e6929, before this comparison existed), so the ONLY new arm64 risk is the sign compare itself. Gate nanExact to the x86 family (NanSignIsX86Reference); NaN tokenizes on arm64 exactly as it did when the tiers were last macOS-green. Same host-pin RULE as the RunHostLibmCorpus tiers, applied per-op. Windows/Linux stay the strict gate. 3. docs job — benchmark/coverage/generated was stale audit_coverage.py --check failed (it was hidden behind the coverage/inventory diffs fixed in 8e1f3cd). Its coverage derives from coverage/generated/coverage.json (the file refreshed in 8e1f3cd) plus the committed benchmark *.cs, so it was stale for the same reason. Regenerated; --check now current. LF output, sorted, .as_posix() — the same deterministic generator family proven byte-identical across Windows/ubuntu.
…t-file edits The api-coverage docs job diffs the committed test/inventory/generated against a fresh regeneration; report.csv records each test's source line numbers, so the two arch-gate edits in fbd8fae (NewDtypesUnaryTests, FuzzCorpusTests.Kinds) re-drifted it. Regenerated in a clean worktree — byte-identical (sha256) to the ubuntu CI artifact, so the diff step is now zero. This is the last refresh (no further test edits).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NumSharp 0.70.0
It took a while but we are at 85% NumPy API coverage.
This is a huge milestone for the .NET ecosystem as NumSharp grows to matureness.
This branch also delivers integration with NumPy's OpenBLAS backend and full integration with Python giving a new angle of use cases for NumSharp to a point NumSharp is an unmanaged memory and math interop with the python ecosystem. I believe in integration rather than competition thus the large scale of support from PyTorch to Pillow.
OpenBLAS is a rapidly developed ecosystem NumSharp will eventually replace with a simpler version but that requires porting of 100k-300k lines of code to achieve complete mathematical parity. OpenBLAS roughly powers 30% of NumPy's.
Deduplicated highlights from the journey3 branch - one line per feature; iteration commits folded in.
📦 New NuGet Packages
Three optional companion packages ship for the first time, co-versioned with NumSharp 0.70.0 (the two interop packages depend on NumSharp.Core; NumSharp.Build is a build-time development dependency that never enters your dependency graph).
All packages are now published as signed NuGet packages.
dot,matmul,inner,vdot,vecdot,matvec,vecmat,tensordot,multi_dot,matrix_power.solve,inv,det,slogdet,tensorsolve,tensorinv.cholesky,qr,svd,svdvals.eig,eigvals,eigh,eigvalsh.lstsq,pinv,matrix_rank,cond,norm.correlate,convolve.arr.ToNumpy()/arr.ToPython()out;pyObj.AsNDArray()/pyObj.FromArrayLike()in.RegisterCodec()once, then pythonnet's ownobj.ToPython()/pyObj.As<NDArray>()round-trip transparently.Auto(view when possible, else copy),View(share or decline),Copy(always independent).FromArrayLikeimports any PEP 3118 exporter, strided/offset/reversed included; read-only stays non-writeable:[NDScoped]/[NDScopedAsync]deterministic memory reclamation: mark a method and theNDArraytemporaries it drops return to NumSharp's buffer pool the moment it exits, instead of waiting on the finalizer - the source keeps its 100% original body; the scope is woven post-compile into the intermediate assembly.lib/, no dependency entries);dotnet add package NumSharp.BuildwritesPrivateAssets="all"by itself, so installing it changes your build, never your package's dependency graph.asyncmethods, iterators, and non-asyncTask/ValueTaskreturns are woven through their compiler state machines; incremental per-TFM, idempotent (double-weaving impossible), and a strong-named consumer is re-signed with its own key.analyzers/dotnet/cs/, so referencing NumSharp alone reports a wrong or unsupported[NDScoped]target as a build error (NDW002-NDW011, NDW015), nudges on leakedNDArraytemporaries (NDW012), and holds types to the same ownership contract: a class/struct that stores NDArrays (a field or auto-property holding anNDArray, an array/tuple/collection/generic of them, a carrier struct, or another NDArray-owning type - ownership is contagious) must beIDisposable/IAsyncDisposable(NDW016) and must dispose every such member on itsDisposepath (NDW017); an instance of such a disposable is then an owned value for NDW012 (new Holder(a + b);or a never-closednp.nditer(a)warns), andforeachover a producedNDArrayor such an instance is flagged (C# disposes only the enumerator). The runtime-inert[NDBorrowed]attribute (field / property / class / struct) states "this references an array owned elsewhere" and opts out. NumSharp also carries the NDW013 build warning for[NDScoped]used without the weaver installed (the attributes are then inert).-p:SkipNDScopeWeave=truebuilds without weaving (nothing else changes);-p:NDScopeWeaveILVerify=trueadditionally runsdotnet-ilverifyon the woven output.tools/verify_build_package.sh, an 18-step real-consumer nupkg flow (package shapes, weave + incrementality, transitive isolation, re-signing, state machines, the analyzer/weaver error layers, NDW012/NDW013, NDW016/NDW017 +[NDBorrowed], and analyzer-via-NumSharp-alone).📊 Dashboards & Docs
Three living dashboards ship on the documentation site, each generated from the same CI artifacts the release gates run on.
✨ New APIs & Modules
np.random.default_rng- the full modern PCG64Generator, byte-identical streams to NumPy 2.4.2 -e868d8ae(+754b7476,febfbbdd,f491c499).default_rng- entry point (seed /SeedSequence/BitGenerator/PCG64overloads).random,integers,standard_normal,normal,exponential,uniform,standard_gamma,gamma,choice,shuffle,permutation,permuted- the Generator draw surface.random_integers,bytes- the legacy RandomState helpers.np.fft.*- the whole 18-function Fourier module, a pure-managed pocketfft port, bit-exact incl. float32/float16 values -3b9d5cfb,a525e355,4cb91898.fft,ifft,fft2,ifft2,fftn,ifftn- complex forward/inverse (1-D/2-D/N-D).rfft,irfft,rfft2,irfft2,rfftn,irfftn- real-input transforms.hfft,ihfft- Hermitian-symmetric transforms.fftfreq,rfftfreq,fftshift,ifftshift- sample-frequency & shift helpers.np.einsum- Einstein summation, now computing and planning -7d2d7a2f(+d78e07db,b61b0998),bb63ba48.einsum- contracts via the matrix products (rides OpenBLAS when the package is referenced).einsum_path- greedy/optimal contraction planner, byte-exact info string.np.r_/np.c_/np.ix_/np.s_/np.index_exp- the grid & slice-expression DSL, 131/131 bit-exact vs NumPy 2.4.2 -00dfe402(+3c63734d,7eea4f7f,c4e27523).np.ogrid/np.mgrid/np.meshgrid- open-mesh / dense-mesh / coordinate-matrix grid constructors, differential bit-exact vs NumPy 2.4.2 -19feaed2,7f558d05,4e8c3925.NDIterRefengine (37 cases probed side-by-side, all identical) -8bd882b3,7112cbe4.np.nditer,np.ndindex,np.ndenumerate- the boxed iterators, full flag/error parity.np.nested_iters,ndarray.flatiter- nested-loop iterators + a write-through flat iterator.b3505398(+8cad3025).partition,argpartition- kth-element partial sort (value + index).lexsort- indirect stable multi-key sort;sort_complex- real-then-imag complex sort.nanargmax,nanargmin- NaN-aware argmax/argmin.np.take_along_axis- the per-slice gather (theargsort/argmaxinverse), NumPy 2.4.2 parity, 24,000+ fuzz cases bit-exact, ≥1.5× faster on every measured variation -f351600a(+88550d13,7091a3c9).np.select- pick each element from the first choice whose condition is true, NumPy 2.4.2 parity (fused single-pass kernel on the contiguous path) -fc10404d(+42d96a14).np.isin+intersect1d/union1d/setxor1d/setdiff1d- element-wise membership + sorted set algebra, NumPy 2.4.2 parity (1.9-13.5× faster) -bfe952d5(+27632ed5).unique_values,unique_counts,unique_inverse,unique_all, 102/102 bit-exact vs NumPy 2.4.2 across 13 dtypes -bec1c497.np.diagfamily + triangular ops - 13 functions, 165/165 side-by-side parity with NumPy 2.4.2 -27b9b012(+a7782984).diag,diagflat,fill_diagonal- diagonal build & in-place fill.tri,tril,triu- triangular masks & extraction.diag_indices,diag_indices_from,tril_indices,tril_indices_from,triu_indices,triu_indices_from,mask_indices- index generators.np.*products; byte-parity via the OpenBLAS backend when referenced -53d7764f(+81509766,297f883f,74aa5d5a).inner,vdot,vecdot,matvec,vecmat,tensordot,multi_dot,matrix_power.956f3392(+2628c921,d6a50593).poly,roots,polyfit,polyval- construction / fitting / evaluation.polyadd,polysub,polymul,polydiv,polyder,polyint- arithmetic & calculus.poly1d- the polynomial object;vander- Vandermonde matrix.savetxt→loadtxtround-trips -a1920a4a,17a1ff8a(+80a0ed50,d39ff824).np.savetxt,np.loadtxt,np.fromstring.9fa48041(+615f1ee5).arcsinh,arccosh,arctanh- primary ufuncs;asinh,acosh,atanh- Array-API aliases.deviceconformance (CPU shim) -ebba2cbf.ndarray.device,ndarray.to_device, anddevice=onarray/zeros/ones/empty/arange/ ….np.kron,np.cross- Kronecker & cross products -7bcad845,73019dce.np.cov,np.corrcoef- covariance & Pearson correlation -92dc537b,aaf731b2.np.choose- index-into-choices gather -aaa41ef2.np.nancumsum,np.nancumprod- NaN-aware cumulative scans -0370c0aa.np.digitize,np.bincount- bin-index + integer histogram, bit-exact vs NumPy 2.4.2 -f2cefba2,12f484c3.np.correlate- sliding cross-correlation (managed SIMD; OpenBLAS byte-parity below) -12f484c3.np.bmat- block-matrix assembly -6ba24752(+d5621d57).np.real,np.imag,np.angle,np.conjugate/np.conj- complex component / phase accessors (post-FFT spectrum extractors) -8b0ac701,d0081b6d.np.iterable- NumPy's pure iterability predicate -ce560796(+8cf54d35).np.isfortran- F-contiguity predicate (a.flags.fnc) -30453696.np.logaddexp,np.logaddexp2,np.nextafter,np.copysign- IEEE binary ufuncs (fullout=/where=/dtype=surface);nextafter/copysignbit-exact,logaddexp≤2 ULP vs NumPy 2.4.2 -043370e0,26d014ed.np.interp- 1-D linear interpolation (incl.period+ complexfp), bit-exact vs NumPy 2.4.2 -043370e0.np.nan_to_num,np.isposinf,np.isneginf- NaN/±inf replacement + signed-infinity predicates, byte-identical to NumPy 2.4.2 -480c2786.np.getbufsize,np.setbufsize- thread-local ufunc buffer size with NumPy's verbatim validation, byte-exact -6d471cf3.np.linalgfactorisation surface and complex128dot/matmulare listed under New NuGet Packages above (they compute via the OpenBLAS backend) -dc448acc,f5ec6276,d09e4376,6ee562da.NDScope- deterministic buffer reclamation:using (var s = NDScope.Open())returns theNDArraytemporaries built inside the scope to the pool at exit (vias.Returns(result)) instead of waiting on the finalizer; Core weaves ~265np.*methods with[NDScoped]so their transients are reclaimed eagerly, and the NumSharp.Build weaver applies the same to your own methods -1b4e776b(+99583e25,726ec48b).🧩 ndarray surface
ndarraymember parity with NumPy 2.4.2 -data- the memoryview buffer object (np.MemoryView); accepted zero-copy byarray/asarray/frombuffer/ … -25ae7053(+4072577d,bc544403).byteswap- width-dispatched endian byte-swap -67994cbc.getfield,setfield- byte-field views -4b07b71d.real,imag,conj,conjugate- complex accessors -7765ce50.itemsize,nbytes,fill,flags- metadata members -792a9f14(+aee7cbab,27a19ae4);setflags- write/align control -275f089c.all,any,clip,take,repeat,squeeze,trace, …) -06869352.⚡ Performance
Ratios are NumPy ÷ NumSharp - higher is better (
x2= twice NumPy's speed);xLOW->xHIGHspans the worst→best measured cell across sizes and dtypes.x0.98->x74-np.uniquefamily routed through the radix sort core -5df10897(+35d12699).x1.6->x5.2-percentile/median/quantilepivot-stack block-partition quickselect -8a1376ff.x0.4->x2.75-np.argpartitionon the same block/pivot-stack path -75a1d873.x1.35->x11-np.isinhash-set membership replaces sort+searchsorted -bd96d541.x15.6- blocked GEBP double GEMM for transposed-Bdot(2.9→43 GFLOP/s) -97e9e82a(+7d680eb1).x40->x249- typednp.nditer<T>/nditer_chunks<T>, allocation-free iteration (chunks +Vector<T>hits 249×) -d58f3728.x1.0->x4.5-take/put/placeelement-copy specialization + gather prefetch (takewent from x0.68 losing to winning everywhere) -88550d13.x1.04->x11.7- float32exp/log/sin/cos/tanh+rad2deg/deg2radreimplemented as bit-exact NumPy kernel ports (tanhalso replaces the float64 loop) -ecdb4581,6bab5754,f5f21ff3.x1.8->x60- the whole float16 family on bit-level AVX2 / widen-compute-narrow kernels:min/max/ptp,maximum/minimum/fmax/fmin, the six comparisons,nanmin/nanmax,clip,add/subtract/multiply/divide,floor/ceil/trunc/rint-a05b5b1e,f3405659,3a46cfec,c9e141c5,498a68f0,c8b0573d,c8babf28.x2.0->x3.4- the bool dtype family on byte-lane SIMD (bitwise/logical/comparisons; was x0.38-0.62);sum(bool)is a popcount (x23.6) andargmax(bool)a find-first scan (up to ~60,000x on sparse input) -46dbb9c6,d967d99b(+bd655743).x0.9->x1.9- SIMDisnan/isinf/isfinitefor float32/float64 (was ~x0.10; 1K stays at the small-N alloc floor) -7bd4f380(+4dfe7619).x1.0->x1.65- float32/float64argmax/argminsingle-pass SIMD tournament (was ~x0.15) -48f894ea.x1.82->x3.12- float32exp2SIMD kernel (hybrid double-2^r+ float scale; was ~x0.19) -87a8bb8f.x1.4->x6.5- NDIter 2-D block kernel for narrow strided rows + NumPy-style axis coalescing (narrow rows were x0.4-0.82; a contiguous(250000,4)array times a scalar dropped 1.7 ms->251 µs) -af25a746,ccadeef4.x1.1->x10.5- fancy indexing (a[idx],a[idx]=v,m[ridx]) routed to the take/put kernels andwhere=masked ops scanned with SIMD (a[idx64]was x0.22,a[idx32]x0.81; masked all-false 27.7->1.5 µs) -09d1fc59.x1.06->x9.3- NDIter fixed-cost cut (recycled state block, packed kernel key, direct external-loop advance, SIMD comparisonout=, eager overlap-temp dispose): construction geomean x2.73->x5.8, everyout=ufunc now beats NumPy at n=1 -15154b00.x2.88->x3.00-cov/corrcoefvia a managed symmetric-Gram (syrk) path at ≤16 variables (100K; was ~x0.56) -d64f3df5.x0.7->x1.85- managedmatvec/vecmat/ matrix-vectordotgemv/gevm paths, no backend (was ~x0.1-0.2) -4f666fc8.x6-diff/ediff1dfused adjacent-difference stencil (1K; ~4x fewer allocations at 100K) -6e94dbcc.x1.46->x4.8-fill_diagonal/diag/diagflatdiagonal-write IL kernel (fill_diagonal10M was x0.40) -a13238ac.x1.4->x1.7- streamed int64/uint64meanaxis reductions + unrolled flatnanmin/nanmax(were x0.16 / x0.26) -8c09dc15.x2.0->x2.3- pre-statecpblkfast path for trivial same-layoutcopyto/copy/cloneat small N -7bfc27a2.x1.8->x2.3- small same-dtype single-broadcast ops routed to the direct SimdChunk kernel (1K; was ~x0.55) -a3819869.x0.6->x1.35- buffer-pool GC pacing + burst-sized buckets lift the small-N elementwise floor for undisposed results (1K float32abswas x0.29) -160ecbba.1088 B -> 192 Bper-NDArrayobject base size (896 B smaller) -UnmanagedStorage's 15 per-dtype slice fields collapsed into oneStructLayout.Explicitunion -8306fa63.🎯 Parity & Fixes
ndarray.flags/setflags- full NumPy 2.4.2 parity across the whole layout/producer space, hardened by a 1104-case differential oracle (owndata/writeable/contiguity, squeeze-as-view, split-child contiguity, read-only reduction scalars) -275f089c,53b5d82e,ca1b0fac.searchsorted- complex lexicographic order +result_typekey promotion (no more silent key down-cast) + NaN-as-largest total order -93abe13d,cc676ea8,f2cefba2.np.take/np.putindex validation matches NumPy - a negative index undermode='raise'normalizes once (np.take(a, [-1])addresses the last element instead of throwing), and a non-castable float/complex index raises the verbatimTypeErrorinstead of silently truncating -fc10404d,88550d13.np.correlate/np.convolve- OpenBLAS byte-parity via the new sliding-dot seam -d0be3132.broadcast_tois read-only,broadcast_arraysis writeable, and writing a non-writeable view now raises NumPy's verbatim message instead of silently corrupting the shared source -1eadb83b,6fb518c0,1cc67d47(+baf41c89).size×itemsizeoverflow,reshape(-1, …), andexpand_dimsaxis now raise NumPy's verbatim texts instead of silent wrong-size allocations or raw .NET exceptions -c2552d6a.ones(3000)@ones(3000)=2048 saturation), stacked/fancy indexing into zero-sized arrays, and the 0-d boolean setter -03d0f0c8,7636100a,f6e258c0.SetIndicesNDNonLinear), bit-exact vs NumPy 2.4.2 across all 15 dtypes -ff68bf14.astype(copy: false)never mutates the caller's array on a dtype conversion, matching NumPy -e5274cdc.ndarray.view(dtype)of a different-itemsize dtype now follows NumPy 2.x's last-axis-contiguous rule, soarr[::2].view(int32)works instead of throwing -970ee7f1.np.matmulgains the full ufunc keyword surface (out=/axes=/axis=/keepdims=/dtype=/casting=/order=), andnp.dot/np.outergainout=-73019dce(+87ff5797).argmax/argminbugs the sort audit exposed - the Decimal and Char flat paths and a NaN-tie ordering - now match NumPy 2.4.2 -8cad3025.np.uniquefull-parameter parity with NumPy 2.4.2 - the axis path's slab equality is corrected so each NaN sub-array is distinct and signed-zero sub-arrays collapse (a real unique-row-count bug for floats/complex),sorted=/equal_nan=are accepted, an out-of-range axis raises the verbatimAxisError, the bare-return overloads (np.unique(ar, axis: 0)) andintersect1d(return_indices:)now port verbatim, andUniqueResultfields are case-identical to NumPy -9f573dd5,262eefd7,0151a832.np.linalgfactorisations without a backend now raise a typedOpenBlasMissingBackendException- derives fromNotSupportedExceptionso existing catches still work, and names theNumSharp.Interop.OpenBLASpackage to install (was a bareNotSupportedException) -d1347c36.b2a8374b:np.ascontiguousarray/np.asfortranarray- a 0-D input returns a length-1 view (shares storage), matching NumPy's ndim≥1 contract.np.eye/np.ones-Charfills numeric one U+0001, not the character'1'.np.full_like- preserves the source array's dtype;fill_value's CLR type no longer selects the result dtype.np.linspace- floors inexact values before an integer-dtype cast and pins the endpoint tostopexactly.np.einsum- a scalar (ndim==0) contraction keeps its()shape instead of promoting to(1,).np.angle(deg: true)- a 0-DHalf/Singleresult keeps its float tier instead of promoting toDouble.var/std/cumsum/cumprod/all/any) andndarray.fillnow run at unlimited ndim like the rest of NumSharp -8f34e8ff,7fa96750.np.linalgfactorisations -det,slogdet,solve,inv(+tensorinv/tensorsolve/matrix_power(n<0)) - now compute in a backend-free Core via a managed LU (allcloseto NumPy, faster for small matrices) instead of raising; an installed OpenBLAS backend still wins the seam for byte-parity -48b00e00(+03884ee9).np.isclose/np.allclosenow compute in NumPy's exactresult_type- fixes a complex128 correctness bug (the imaginary part was dropped, soisclose([1+0j],[1+100j])wrongly returnedTrue) and evaluates float32 pairs in float32 (100K x0.18->x3.3) -fbbda5f0.np.sum(float16)accumulates in a float32 shadow and narrows per orientation like NumPy'sHALF_add- an axis sum now saturates (sum(ones((4096,3),f16),axis=0)=[2048,2048,2048], was[4096,...], a ~3.5% error) while a flat sum still reaches 4096 -32732a0f.np.power(x, negative_int)- a strided/2-D/broadcast integer exponent no longer reads out of bounds (a Release memory-safety bug), and a bool base now raises NumPy's verbatimValueErrorinstead of silently computing -02e6929f.new NDArray(buffer, shape, 'F')lays out column-major,np.arange(dtype=bool)raises past length 2,np.frombufferrejects complex64/'c8', andnp.array's defaultndminis 0 -5c7e3ad8.np.isreal/np.iscomplexnow inspect the imaginary part (they returned all-True/ all-Falsefor complex regardless of value) and no longer emit garbage bytes on a strided real input -fa491573.sum/mean/prod/min/max/std/var+ allnan*) preserve an F-contiguous input's layout (KEEPORDER allocation) instead of flipping it to C, matching NumPy -0ae977d9(issue [Core] Layout 'F/A/K' support #610).np.clipon a non-contiguous Boolean array (strided/transposed/F-order/reversed) now clips instead of throwingNotSupportedException-f6f5b657.add/subtract/multiply/divide(andkron) no longer read the wrong elements - a stride-coalescer bug (merged adjacent axes by value, not magnitude) plus a bit-exact odometer kernel -7f2c09a3.maximum/minimum/nanmin/nanmax/clipnow return NaN operands verbatim (payload + sign preserved, was canonicalHalf.NaN), and aclipNaN-max-bound precedence bug that returned float32 fills (any dtype) is fixed -f3405659,498a68f0,c9e141c5.🧰 Testing & Tooling
out=/where=(3,727 cases over out × mask layouts), result-kinds + verbatim-error + iterator-trace, IEEE special-values (nan/±inf/±0/subnormal), and a truthful-vs-precise precision channel -bc91dd25,6cd1de9b,0882edbb,359e9d3c,76f0c918.np.randomsampler byte-parity divergences (f/pareto/standard_cauchy/binomial/negative_binomial/multinomial/multivariate_normal/gamma(shape<1)), now pinned as known[OpenBugs]issues (not yet fixed) -31a178f2.cf559a1a,03415ec9,5ff54a72.💥 Breaking Changes
np.random.bytes/Generator.bytesnow returnNDArray<byte>instead ofbyte[], so draws >2 GiB succeed (NumPynpy_intpparity) -44d2e7d9.ndarray.stridesnow reports bytes per axis (was elements), matching NumPy'sPyArray_STRIDES-6ef30215.np.unique(ar)now returns aUniqueResultstruct instead of a bareNDArray, sonp.unique(ar)[k]selects the k-th output (use.values[k]for the k-th value); it converts implicitly toNDArray/NDArray[]so most call-sites are unchanged -17f571ef.np.mgrid/np.meshgriddrop their legacy non-NumPy signatures:mgrid[...]is now an indexer (was a 2-arg method) andmeshgridis variadic returningMeshgridResult(was a fixed 2-tuple +Kwargs) -7f558d05,4e8c3925.PublicKeyTokenchanges fromnulltocc7b13ffcd2ddd51(published NumSharp had shipped unsigned since 2019); every consumer (TensorFlow.NET, Pandas.NET, Gym.NET) must recompile -478d550d.NUMSHARP_<AREA>_<SETTING>scheme with no back-compat aliases -NUMSHARP_GUARD_PAGES(shipped in 0.60.0) becomesNUMSHARP_DEBUG_GUARD_PAGES, and the OpenBLAS/pythonnet knobs take_LIBRARY/_SEARCH_PATH/_USE_BUNDLED/_PYPI_FEED_URL/_REQUIRE_ENGINEnames -079d1859.NDArray.Normalize()(a non-NumPy extension) is marked[Obsolete]in favour ofnp.clip()-b701843e.boolarray combined with a weak integer literal now promotes to int64 (was int32), matching NumPy's NEP50 -np.left_shift(boolArr, 2),boolArr + 2,boolArr & 2etc. change result dtype; a narrower strong spelling like(short)2keeps its own kind -46dbb9c6.np.poly1dis nowIDisposable- it owns the coefficient array its constructor yields into it, soDispose()releases it, and the copy-constructornew poly1d(p)now copies the coefficients instead of sharing one array between two owners (NumPy shares by refcount; two NumSharp owners would double-dispose). Found by the new NDW016 ownership analyzer, which also madenp.Broadcast.Dispose()(whatforeachcalls when a loop ends) release thebroadcast_toviews it had built (they are rebuilt lazily on the nextiters/enumeration, so the object stays re-enumerable) andIndexCollectoranIDisposablethat no longer strands its outgrown buffer -4cc9cac5.