diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 02b3ab0..f787f30 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -57,7 +57,7 @@ jobs: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: - go-version: 1.27.1 + go-version: 1.27.2 - run: go version - run: go vet ./... - run: go build ./... @@ -89,7 +89,7 @@ jobs: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: - go-version: 1.27.1 + go-version: 1.27.2 - run: go env GOARCH - name: Regenerate committed assembly and check it is current run: | @@ -126,7 +126,7 @@ jobs: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: - go-version: 1.27.1 + go-version: 1.27.2 - run: sudo apt-get update && sudo apt-get install -y qemu-user-static - name: Regenerate committed assembly and check it is current if: matrix.goarch == 'ppc64le' || matrix.goarch == 'loong64' || matrix.goarch == 'riscv64' || matrix.goarch == 's390x' @@ -150,7 +150,7 @@ jobs: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: - go-version: 1.27.1 + go-version: 1.27.2 - name: Vet + test (GOARCH=386) run: | GOARCH=386 go vet ./... @@ -167,7 +167,7 @@ jobs: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: - go-version: 1.27.1 + go-version: 1.27.2 - name: go test -c (go vet on android/ios) for each go tool dist list pair run: | fail=0 diff --git a/README.md b/README.md index 0882180..4c0a8ab 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,9 @@ # ndarray — go-ndarray [![Docs](https://img.shields.io/badge/docs-mkdocs--material-013243)](https://go-ndarray.github.io/docs/) +[![Playground](https://img.shields.io/badge/playground-try%20it%20in%20your%20browser-013243)](https://go-ndarray.github.io/playground/) [![License](https://img.shields.io/badge/license-BSD--3--Clause-blue)](LICENSE) -[![Go](https://img.shields.io/badge/go-1.27.1%2B-00ADD8)](https://go.dev/dl/) +[![Go](https://img.shields.io/badge/go-1.27.2%2B-00ADD8)](https://go.dev/dl/) [![Status](https://img.shields.io/badge/status-numpy%20parity%20(float64)-9a6700)](docs/plan-ndarray.md) **A pure-Go (CGO=0) NumPy-style N-dimensional array library.** Row-major @@ -30,6 +31,9 @@ `ws.Use`, compute as usual, `ws.Reset()` at the end of each pass. Results stop costing the garbage collector anything (see below). +**Try it without installing anything:** the [playground](https://go-ndarray.github.io/playground/) +runs go-ndarray compiled to WebAssembly in your browser. + ### Loops: `Workspace` Every operation returns a new array, and in a loop those results are garbage @@ -66,9 +70,8 @@ arm64, VSX on ppc64le, LASX on loong64, RVV on riscv64 and the vector facility on s390x, the last three when the CPU has them), max/min too on amd64 and arm64, and a **panel-packed, cache-blocked GEMM** with an SIMD-FMA micro-kernel (NEON 4×8; AVX2/FMA 6×8 chosen at run time, SSE2 fallback; VSX -8×8; LASX 8×8; RVV 6×8), the OpenBLAS/BLIS structure. The 32-bit targets run -the same pure-Go code those kernels are tested against. The s390x kernels are -validated under emulation only (no s390x host was reachable to measure them). `Exp` and `Log` are ports of Arm's optimized-routines +8×8; LASX 8×8; RVV 6×8; s390x 4×8), the OpenBLAS/BLIS structure. The 32-bit +targets run the same pure-Go code those kernels are tested against. `Exp` and `Log` are ports of Arm's optimized-routines (≤ 0.51 ULP measured, against up to 0.88 and 0.72 for Go's `math.Exp`/`math.Log` on arm64), and they are correct where amd64's `math.Exp` returns +Inf diff --git a/SECURITY.md b/SECURITY.md index 934a103..f77aea9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -57,6 +57,28 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none. ## Audit log +**2026-10-09 (v0.9.0).** + +- **The Go floor is 1.27.2.** At 1.27.1, `govulncheck` listed standard-library + vulnerabilities fixed in 1.27.2 (GO-2026-6604, -6612, -6613, -6617, among + others), none of them reachable from this library. At 1.27.2 it reports none, + for the library or for its generator module. +- **The loong64 miscompile is not fixed in 1.27.2.** An earlier entry said + golang/go#81000 was fixed in 1.27.2; it was not. The backport + (golang/go#81147) is on the 1.27.3 milestone. On real hardware (cfarm401, + Loongson 3C5000L), the upstream reproducer (`go-gfx/gfx` `./color`, + `TestSkimageLabToRGBOutOfGamut`) **FAILS** when built with go1.27.1 and with + go1.27.2 alike. The evidence that this library does not hit the pattern + stands as before: the suite and `FuzzOps` on that machine. Raise the floor + to 1.27.3 when it ships. +- **The s390x kernels now run on real hardware**: an IBM z17, through the + LinuxONE Community Cloud (`/proc/sysinfo`: Type 9175, a z/VM 7.3.0 guest + with 8 vCPUs in a shared LPAR). The whole suite and the fence tests pass, + and `sumVX` reading one element past the end faults (4 KiB pages). The new + 4×8 GEMM micro-kernel is held to the same checks: it is guarded by its + wrapper and fenced, and a sabotaged copy that drops tile row 3 fails + (`MatMul m=50 k=70 n=90 [270]: 0 != -249`). + **2026-10-08 (v0.8.0).** New s390x vector-facility kernels, held to the same guards (wrappers check each kernel's reach; the PROT_NONE fence tests now include s390x). ⚠ They were validated **under emulation only** (Docker's @@ -114,7 +136,9 @@ two defects. Both are fixed, each with a test that fails without the fix. - go1.27.0 and go1.27.1 miscompile one loong64 pattern ([golang/go#81000](https://github.com/golang/go/issues/81000): a byte-sized inlined result spilled with an 8-byte store, clobbering the stack slot next - to it), fixed in 1.27.2. On a real Loongson 3C5000L with LASX (cfarm401), + to it), ~~fixed in 1.27.2~~ — **wrong, corrected 2026-10-09**: the backport + (golang/go#81147) moved to the 1.27.3 milestone, and go1.27.2 still fails + the upstream reproducer on cfarm401 (see the v0.9.0 entry). On a real Loongson 3C5000L with LASX (cfarm401), built with go1.27.1, the whole suite passes, including the fence tests, and `FuzzOps` ran 52.7 million executions natively without a divergence from its oracle. That is evidence that this library does not hit the pattern, not diff --git a/docs/perf.md b/docs/perf.md index 3f9acdd..4d27baa 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -878,15 +878,37 @@ stay dynamic, so its slower efficiency cores are handled as before. - **s390x (vector facility)**, since v0.8.0, when AT_HWCAP reports it: sum, dot, sqrt and add/sub/mul/div, eight float64 per loop in four two-lane registers plus a scalar tail, every instruction a Go mnemonic. - max/min and the GEMM micro-kernel stay scalar (the vector max with NumPy's - NaN rule needs the z14, beyond the z13 Go supports). The operand order of + Since v0.9.0 the GEMM micro-kernel is a 4×8 tile: sixteen two-lane + accumulators, the B row in one `VLM`, each A value replicated with `VLREPG` + and fused in with `VFMADB`. max/min stay scalar: the vector max with NumPy's + NaN rule needs the z14, beyond the z13 Go supports. The operand order of `VFSDB`/`VFDDB`/`VFMADB` was **measured**, not read: each run once on known - inputs (`VFSDB V1, V2, V3` is V3 = V2 − V1). ⚠ **Validated under emulation - only** (Docker's qemu-s390x locally, qemu-s390x in CI): the whole suite, the - fence tests, and two sabotaged kernels that both failed. **Not measured**: - the LinuxONE host did not answer. AT_HWCAP is read in the machine's byte - order, which on big-endian s390x is not the little-endian the parser first - assumed. + inputs (`VFSDB V1, V2, V3` is V3 = V2 − V1). AT_HWCAP is read in the + machine's byte order, which on big-endian s390x is not the little-endian the + parser first assumed. + + v0.8.0 shipped these kernels validated under emulation only. Since then they + run on **real hardware**: an IBM z17 (Type 9175), through the LinuxONE + Community Cloud, as a z/VM 7.3.0 guest with 8 vCPUs in a shared LPAR. There + the whole suite passes, as do the fence tests (a kernel reading one element + too far faults), `TestParallelStress` (27,752 calls) and two sabotaged GEMM + and elementwise kernels, which fail as they should. Measured with `hostload` + (0% busy, 0% steal) and `benchstat`, 8 interleaved runs each, every + difference at p = 0.000 unless marked: + + | IBM z17, time vs the scalar code | 1 core | 8 cores | + |---|---|---| + | `Sum` | −85% to −87% | −82% to −87% | + | `AddInto` / `MulInto` | −44% to −61% | −36% to −60% | + | `DivInto` | −6% to −18% | −6% to −22% | + | `SqrtInto` | −2% to −11% | −5% to −9% at 1–16 Ki, ~ above | + | `MatVec` 1024² | −44% | −40% | + | `Dot` 2^20 | −11% | ~ (p = 0.08) | + | `MatMul` 1024² (v0.9.0 GEMM) | 340 → 70 ms (−79%) | 43 → 10 ms (−76%) | + | GEMM geomean, 11 shapes | −74% | −66% | + + `Exp`, which none of this touches, measured within ±0.3% on one core: the + control. - The 32-bit targets keep the validated scalar oracles, using the same four-accumulator max/min, direct sqrt loop, and a **scalar 4×4 GEMM micro-kernel** over the packed panels, and diff --git a/docs/plan-ndarray.md b/docs/plan-ndarray.md index 199d471..f85b30e 100644 --- a/docs/plan-ndarray.md +++ b/docs/plan-ndarray.md @@ -127,7 +127,8 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`). reports LASX), and since v0.7.0 riscv64 has RVV kernels (sum, dot, sqrt, elementwise, and since v0.7.1 a 6×8 GEMM micro-kernel; used when the CPU reports V), and since v0.8.0 s390x has vector-facility kernels (sum, dot, - sqrt, elementwise; validated under emulation, not yet measured). The + sqrt, elementwise, and since v0.9.0 a 4×8 GEMM micro-kernel; measured on a + real IBM z17). The 32-bit targets keep the validated scalar oracles plus a scalar 4×4 GEMM micro-kernel over the packed panels, with packing + cache-blocking + multicore; they have not diff --git a/go.mod b/go.mod index e521f5f..94f5256 100644 --- a/go.mod +++ b/go.mod @@ -1,3 +1,3 @@ module github.com/go-ndarray/ndarray -go 1.27.1 +go 1.27.2 diff --git a/internal/kernels/asmgen/go.mod b/internal/kernels/asmgen/go.mod index 6b52ea1..7a09d65 100644 --- a/internal/kernels/asmgen/go.mod +++ b/internal/kernels/asmgen/go.mod @@ -4,6 +4,6 @@ // go-asmgen to (re)produce the committed .s files via `go generate`. module github.com/go-ndarray/ndarray/internal/kernels/asmgen -go 1.27.1 +go 1.27.2 require github.com/go-asmgen/asmgen v0.16.0 diff --git a/internal/kernels/asmgen/s390x/gen.go b/internal/kernels/asmgen/s390x/gen.go index eb16a84..730d825 100644 --- a/internal/kernels/asmgen/s390x/gen.go +++ b/internal/kernels/asmgen/s390x/gen.go @@ -36,6 +36,7 @@ func main() { } { f.Add(binKernel(k.name, k.vec, k.sca)) } + f.Add(gemmKernel()) if err := os.WriteFile("sum_s390x.s", []byte(f.String()), 0o644); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) @@ -138,3 +139,42 @@ func binKernel(name, vec, sca string) *emit.Function { Raw("done:").Ret() return b.Func() } + +// gemmKernel builds gemmMicro4x8VX(kc int, pa, pb, c *float64, ldc int) for +// kc >= 1: C[r][0:8] += sum_p pa[p*4+r] * pb[p*8:p*8+8] for r < 4, the rows of C +// ldc float64s apart. Sixteen two-lane accumulators V0..V15 (row r in +// V4r..V4r+3); each step loads the B row in one VLM, replicates each A value +// across a register with VLREPG, and fuses it in with VFMADB. +func gemmKernel() *emit.Function { + sig := s390x.Layout([]string{"kc", "pa", "pb", "c", "ldc"}, + []s390x.Type{s390x.Int64, s390x.Ptr, s390x.Ptr, s390x.Ptr, s390x.Int64}, nil, nil) + b := s390x.NewFunc("gemmMicro4x8VX", sig, 0) + b.LoadArg("kc", "R5").LoadArg("pa", "R2").LoadArg("pb", "R3"). + LoadArg("c", "R1").LoadArg("ldc", "R4") + for r := 0; r < 16; r++ { + b.Raw("VZERO V%d", r) + } + b.Raw("loop:").Raw("VLM (R3), V16, V19") // the B row + for r := 0; r < 4; r++ { + b.Raw("VLREPG %d(R2), V%d", 8*r, 20+r) // a_r in both lanes + } + for r := 0; r < 4; r++ { + for j := 0; j < 4; j++ { + b.Raw("VFMADB V%d, V%d, V%d, V%d", 20+r, 16+j, 4*r+j, 4*r+j) // acc += a_r * b + } + } + b.Raw("ADD $32, R2").Raw("ADD $64, R3").Raw("SUB $1, R5"). + Raw("CMP R5, $0").Raw("BNE loop"). + Raw("SLD $3, R4") // row stride in bytes + for r := 0; r < 4; r++ { + b.Raw("VLM (R1), V24, V27") + for j := 0; j < 4; j++ { + b.Raw("VFADB V%d, V%d, V%d", 4*r+j, 24+j, 24+j) // C row += acc_r + } + b.Raw("VSTM V24, V27, (R1)") + if r < 3 { + b.Raw("ADD R4, R1") + } + } + return b.Ret().Func() +} diff --git a/internal/kernels/gemm_generic.go b/internal/kernels/gemm_generic.go index b2b8b07..98f8629 100644 --- a/internal/kernels/gemm_generic.go +++ b/internal/kernels/gemm_generic.go @@ -1,4 +1,4 @@ -//go:build !arm64 && !amd64 && !ppc64le && !loong64 && !riscv64 +//go:build !arm64 && !amd64 && !ppc64le && !loong64 && !riscv64 && !s390x package kernels diff --git a/internal/kernels/gemm_s390x.go b/internal/kernels/gemm_s390x.go new file mode 100644 index 0000000..0a261b1 --- /dev/null +++ b/internal/kernels/gemm_s390x.go @@ -0,0 +1,47 @@ +package kernels + +// s390x GEMM micro-kernel: a 4x8 vector-facility tile (gemmMicro4x8VX in +// sum_s390x.s, generated by go-asmgen): sixteen two-lane accumulators, the B +// row loaded once per step with VLM, each A value replicated with VLREPG and +// fused in with VFMADB. Without the facility, gemmMicroGo computes the same +// tile in scalar code. + +// MR, NR are the micro-kernel's register-block dimensions. +const ( + MR = 4 + NR = 8 +) + +func gemmMicro(kc int, pa, pb, dst []float64, ldc int) { + if useVX { + _, _, _ = pa[kc*MR-1], pb[kc*NR-1], dst[(MR-1)*ldc+NR-1] // the kernel's whole reach + gemmMicro4x8VX(kc, &pa[0], &pb[0], &dst[0], ldc) + return + } + gemmMicroGo(kc, pa, pb, dst, ldc) +} + +// gemmMicroGo is the scalar MR x NR tile: dst[r*ldc+c] += sum_p +// pa[p*MR+r]*pb[p*NR+c], in p order per element. +func gemmMicroGo(kc int, pa, pb, dst []float64, ldc int) { + var acc [MR * NR]float64 + for p := 0; p < kc; p++ { + a := pa[p*MR : p*MR+MR : p*MR+MR] + b := pb[p*NR : p*NR+NR : p*NR+NR] + for r, av := range a { + row := acc[r*NR : r*NR+NR : r*NR+NR] + for c, bv := range b { + row[c] += av * bv + } + } + } + for r := 0; r < MR; r++ { + d := dst[r*ldc : r*ldc+NR : r*ldc+NR] + for c := range d { + d[c] += acc[r*NR+c] + } + } +} + +//go:noescape +func gemmMicro4x8VX(kc int, pa, pb, c *float64, ldc int) diff --git a/internal/kernels/sum_s390x.s b/internal/kernels/sum_s390x.s index 101567a..d9840be 100644 --- a/internal/kernels/sum_s390x.s +++ b/internal/kernels/sum_s390x.s @@ -252,3 +252,82 @@ TEXT ·divVX(SB), NOSPLIT, $0-32 done: RET +TEXT ·gemmMicro4x8VX(SB), NOSPLIT, $0-40 + MOVD kc+0(FP), R5 + MOVD pa+8(FP), R2 + MOVD pb+16(FP), R3 + MOVD c+24(FP), R1 + MOVD ldc+32(FP), R4 + VZERO V0 + VZERO V1 + VZERO V2 + VZERO V3 + VZERO V4 + VZERO V5 + VZERO V6 + VZERO V7 + VZERO V8 + VZERO V9 + VZERO V10 + VZERO V11 + VZERO V12 + VZERO V13 + VZERO V14 + VZERO V15 + loop: + VLM (R3), V16, V19 + VLREPG 0(R2), V20 + VLREPG 8(R2), V21 + VLREPG 16(R2), V22 + VLREPG 24(R2), V23 + VFMADB V20, V16, V0, V0 + VFMADB V20, V17, V1, V1 + VFMADB V20, V18, V2, V2 + VFMADB V20, V19, V3, V3 + VFMADB V21, V16, V4, V4 + VFMADB V21, V17, V5, V5 + VFMADB V21, V18, V6, V6 + VFMADB V21, V19, V7, V7 + VFMADB V22, V16, V8, V8 + VFMADB V22, V17, V9, V9 + VFMADB V22, V18, V10, V10 + VFMADB V22, V19, V11, V11 + VFMADB V23, V16, V12, V12 + VFMADB V23, V17, V13, V13 + VFMADB V23, V18, V14, V14 + VFMADB V23, V19, V15, V15 + ADD $32, R2 + ADD $64, R3 + SUB $1, R5 + CMP R5, $0 + BNE loop + SLD $3, R4 + VLM (R1), V24, V27 + VFADB V0, V24, V24 + VFADB V1, V25, V25 + VFADB V2, V26, V26 + VFADB V3, V27, V27 + VSTM V24, V27, (R1) + ADD R4, R1 + VLM (R1), V24, V27 + VFADB V4, V24, V24 + VFADB V5, V25, V25 + VFADB V6, V26, V26 + VFADB V7, V27, V27 + VSTM V24, V27, (R1) + ADD R4, R1 + VLM (R1), V24, V27 + VFADB V8, V24, V24 + VFADB V9, V25, V25 + VFADB V10, V26, V26 + VFADB V11, V27, V27 + VSTM V24, V27, (R1) + ADD R4, R1 + VLM (R1), V24, V27 + VFADB V12, V24, V24 + VFADB V13, V25, V25 + VFADB V14, V26, V26 + VFADB V15, V27, V27 + VSTM V24, V27, (R1) + RET + diff --git a/internal/kernels/vx_s390x_test.go b/internal/kernels/vx_s390x_test.go index dcd6896..71bc41e 100644 --- a/internal/kernels/vx_s390x_test.go +++ b/internal/kernels/vx_s390x_test.go @@ -12,6 +12,7 @@ func TestVXDetected(t *testing.T) { t.Run("scalar", func(t *testing.T) { TestBinSIMD(t) TestSumSIMD(t) + TestGemmParallelYielding(t) // gemmMicroGo against the oracle }) }