diff --git a/CHANGELOG.md b/CHANGELOG.md index 6abdc611..c0306283 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `main`, the release pipeline automatically replaces `[current]` with the next version number before tagging the release. +## [current] + +### Added + +- **`ae build --size`** compiles with `-Os -g0` (`-Oz` under `--target`) and strips at link + (`-Wl,--strip-all -Wl,--gc-sections`), for a shipped artifact rather than a + debuggable one (#1729). Every other mode pointed at debugging — `--quick` is + `-O0 -g`, `--profile` is `-O2 -g -fno-omit-frame-pointer`, `--coverage` is + `-O0 -g --coverage` — so anyone shipping a library had to emit the C and + hand-compile it. It matters most under `--target`: `zig cc` emits DWARF **by + default** even at `-O2`, and the cross backend passed no `-g0`, so a + cross-compiled `--emit=lib` artifact was overwhelmingly debug information — + measured at **97.4%** of a two-function wasi library, which `--size` takes + from 956,573 to 24,942 bytes, a **38×** reduction with identical behaviour. + The equivalent native `.so` has zero `.debug*` sections, so this was a + cross-path problem rather than something every target shipped; native still + gains about 14%. Deliberately not the default, and deliberately not applied + to `--emit=obj`/`--emit=csrc`, whose symbols are what the next linker needs. + ## [0.577.0] ### Fixed diff --git a/docs/build-system.md b/docs/build-system.md index 8d86b1b6..837f526e 100644 --- a/docs/build-system.md +++ b/docs/build-system.md @@ -130,6 +130,55 @@ and adds only what the profiler needs to attribute it. It works under `--coverage` takes precedence if both are passed: gcov's line attribution needs `-O0`, which is a correctness requirement rather than a preference. +### `--size` for shipped artifacts + +`--size` compiles with `-Os -g0` (`-Oz` under `--target`) and strips at link +time — `-Wl,--strip-all -Wl,--gc-sections` with GNU ld and LLD, +`-Wl,-x -Wl,-dead_strip` on Apple targets, whose linker rejects the GNU +spellings as unknown options: + +```sh +ae build --target=wasm32-wasi --emit=lib mylib.ae -o lib.wasm # 956,573 bytes +ae build --target=wasm32-wasi --emit=lib --size mylib.ae -o lib.wasm # 24,942 bytes +``` + +Every other mode points at debugging — `--quick` is `-O0 -g`, `--profile` is +`-O2 -g -fno-omit-frame-pointer`, `--coverage` is `-O0 -g --coverage` — and the +default `-O2` sits between them. `--size` is the one that points at shipping. + +**It matters most under `--target`.** `zig cc` emits DWARF **by default**, even +at `-O2`, and the cross backend passed no `-g0` — so a cross-compiled +`--emit=lib` artifact was overwhelmingly debug information. Measured on a +two-function wasi library, 97.4% of the module was `.debug*`/`name` sections; +code and data were the remaining 2.6%. The equivalent native `.so` has **zero** +`.debug*` sections, so this was a cross-path problem rather than something +every target shipped. Native builds still benefit, just far less: about 14% on +the same library, from `-Oz` and the symbol table. + +Stripping is behaviour-preserving. The 24 KB module above still instantiates, +still exports every symbol, and still runs identically — including the +fail-stop panic path WASI has. + +`-Os` rather than `-Oz` on the native path: gcc only gained `-Oz` in GCC 12, +and Ubuntu 22.04 — the CI baseline — ships GCC 11, where it is a hard error. +`-Os` is supported everywhere and gives nearly the same result. Cross builds do +use `-Oz`, because zig bundles its own clang and the version is not the host's +to vary. + +Two things `--size` deliberately does not do: + +- **It is not the default.** A 38× difference is discoverable; anyone shipping + to a browser will find the flag. Stripping every build by default would make + the first "why can't I get a stack trace from my wasm module" report + genuinely hard to diagnose. +- **It does not strip `--emit=obj` or `--emit=csrc`.** Neither links, and an + object file's symbols are exactly what whoever links it next needs. Those + modes still get `-Oz -g0` for the compile, but no link-time stripping. + +`--profile` takes precedence if both are passed: asking for a small artifact +and a profileable one is contradictory, and the debug-oriented reading is the +safer one. + ### Resolving the build target `ae build` accepts either a path to a `.ae` file or a `[[bin]]` name from `aether.toml`. The two are equivalent: diff --git a/tests/ae_sweep_prune.txt b/tests/ae_sweep_prune.txt index 490b7b93..177c96b2 100644 --- a/tests/ae_sweep_prune.txt +++ b/tests/ae_sweep_prune.txt @@ -157,6 +157,7 @@ tests/integration/sealed_namespaces/ tests/integration/selective_import_alias/ tests/integration/selective_import_merge_order/ tests/integration/selective_import_shadow/ +tests/integration/size_mode/ tests/integration/source_location/ tests/integration/source_location_default_capture/ tests/integration/spec_format_reporting/ diff --git a/tests/integration/size_mode/sizelib.ae b/tests/integration/size_mode/sizelib.ae new file mode 100644 index 00000000..fd6f5089 --- /dev/null +++ b/tests/integration/size_mode/sizelib.ae @@ -0,0 +1,18 @@ +// Library-shaped source for the --size test. Uses try/catch/panic so the +// panic machinery is linked in too -- a fixture that pulls in more of the +// runtime is a better size signal than one that pulls in almost none. + +risky(x: int) -> int { + if x < 0 { + panic("negative") + } + return x * 2 +} + +safe(x: int) -> int { + try { + return risky(x) + } catch reason { + return 0 + } +} diff --git a/tests/integration/size_mode/test_size_mode.sh b/tests/integration/size_mode/test_size_mode.sh new file mode 100755 index 00000000..5ac7fc17 --- /dev/null +++ b/tests/integration/size_mode/test_size_mode.sh @@ -0,0 +1,144 @@ +#!/bin/sh +# `ae build --size` produces a smaller artifact without breaking it. +# +# ae build had --quick (-O0 -g), --profile (-O2 -g -fno-omit-frame-pointer) +# and --coverage (-O0 -g --coverage) -- all debug-oriented -- and no mode +# pointing the other way. That mattered most on the cross path: `zig cc` +# emits DWARF by DEFAULT even at -O2, and nothing passed -g0, so a +# cross-compiled --emit=lib artifact was overwhelmingly debug information. +# +# Asserts, in rising order of what would actually break a user: +# - --size is accepted and the binary still runs (native exe) +# - --emit=lib under --size keeps its dynamic symbols (a stripped library +# with no symbols links fine and is useless) +# - --emit=obj under --size keeps its symbols (stripping an object would +# remove what whoever links it next needs) +# - the cross wasm artifact is dramatically smaller AND still valid +# +# The cross half is skipped without zig. Cost: ONE cross link (~90 TUs; no +# per-target archive cache), matching the convention in the neighbouring +# cross tests. + +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" +AE="$ROOT/build/ae" + +if [ ! -x "$AE" ]; then + echo " [SKIP] size_mode: ae not built" + exit 0 +fi + +TMPDIR_T="$(mktemp -d)" +cleanup() { rm -rf "$TMPDIR_T"; } +trap cleanup EXIT + +LIB="$SCRIPT_DIR/sizelib.ae" + +# ---- 1. a --size executable still runs ----------------------------------- +cat > "$TMPDIR_T/app.ae" <<'AEEOF' +main() { + println("size mode ok") +} +AEEOF +# Keep the compiler's own message: a bad flag (say, one the host gcc is too +# old for) is invisible if this is swallowed, and "--size build failed" alone +# sends you looking in the wrong place. +if ! "$AE" build --size "$TMPDIR_T/app.ae" -o "$TMPDIR_T/app" \ + >"$TMPDIR_T/build.log" 2>&1; then + echo " [FAIL] size_mode: --size build failed" + sed -n '1,15p' "$TMPDIR_T/build.log" | sed 's/^/ /' + exit 1 +fi +OUT=$("$TMPDIR_T/app" 2>&1) || { echo " [FAIL] size_mode: --size binary did not run"; exit 1; } +[ "$OUT" = "size mode ok" ] || { + echo " [FAIL] size_mode: wrong output: $OUT"; exit 1; } + +# ---- 2. --emit=lib keeps the symbols a consumer resolves against --------- +# Stripping removes the STATIC symbol table but must leave the dynamic / +# external ones; a library with neither would satisfy a size check and be +# unusable. +# +# The listing command is not portable. GNU nm spells it `nm -D`; BSD/macOS nm +# has no -D at all and spells defined-external `nm -gU`. Getting this wrong +# is not a harmless mismatch: the unsupported form exits non-zero with empty +# output, which reads exactly like "the symbols are gone" and fails the test +# on a perfectly good library. So pick by platform, and SKIP rather than fail +# if neither form works -- an inconclusive probe must not masquerade as a +# regression. +if "$AE" build --emit=lib --size "$LIB" -o "$TMPDIR_T/lib.so" >/dev/null 2>&1; then + SYMS="" + case "$(uname -s 2>/dev/null)" in + Darwin) SYMS=$(nm -gU "$TMPDIR_T/lib.so" 2>/dev/null || true) ;; + *) SYMS=$(nm -D --defined-only "$TMPDIR_T/lib.so" 2>/dev/null || true) ;; + esac + if [ -n "$SYMS" ]; then + if ! printf '%s' "$SYMS" | grep -q 'safe'; then + echo " [FAIL] size_mode: --emit=lib --size dropped the exported symbols" + echo " (listing was non-empty, so this is a real strip, not a" + echo " missing nm option)" + exit 1 + fi + fi +fi + +# ---- 3. --emit=obj keeps its symbols ------------------------------------- +# An object file is linked later by someone else, so stripping it would +# remove exactly what they need. --size must not apply link-time stripping +# to a mode that does not link. +# Plain `nm` (no -D) works on both toolchains for an object file, but the +# same "empty means inconclusive" rule applies. +if "$AE" build --emit=obj --size "$LIB" -o "$TMPDIR_T/lib.o" >/dev/null 2>&1; then + OSYMS=$(nm "$TMPDIR_T/lib.o" 2>/dev/null || true) + if [ -n "$OSYMS" ]; then + if ! printf '%s' "$OSYMS" | grep -q 'safe'; then + echo " [FAIL] size_mode: --emit=obj --size stripped the object's symbols" + exit 1 + fi + fi +fi + +# ---- 4. the cross case, which is what this mode is for ------------------- +if ! command -v zig >/dev/null 2>&1; then + echo " [PASS] size_mode: native checks (cross skipped: zig not on PATH)" + exit 0 +fi + +BASE="$TMPDIR_T/base.wasm" +SIZED="$TMPDIR_T/sized.wasm" +"$AE" build --target=wasm32-wasi --emit=lib "$LIB" -o "$BASE" >/dev/null 2>&1 \ + || { echo " [FAIL] size_mode: baseline wasm build failed"; exit 1; } +"$AE" build --target=wasm32-wasi --emit=lib --size "$LIB" -o "$SIZED" >/dev/null 2>&1 \ + || { echo " [FAIL] size_mode: --size wasm build failed"; exit 1; } + +BASE_SZ=$(wc -c < "$BASE" | tr -d '[:space:]') +SIZED_SZ=$(wc -c < "$SIZED" | tr -d '[:space:]') + +# The measured ratio is ~38x; assert a conservative 4x so a partial +# regression (say, -g0 lost but stripping kept) still trips this. +if [ "$SIZED_SZ" -ge $((BASE_SZ / 4)) ]; then + echo " [FAIL] size_mode: --size wasm not meaningfully smaller" + echo " baseline=$BASE_SZ sized=$SIZED_SZ (wanted < baseline/4)" + exit 1 +fi + +# Smaller is worthless if it is no longer a wasm module. +case "$(file -b "$SIZED" 2>/dev/null)" in + *WebAssembly*) ;; + *) + echo " [FAIL] size_mode: --size output is not a wasm module:" + echo " $(file -b "$SIZED" 2>/dev/null)" + exit 1 + ;; +esac + +# ...and worthless again if the exports it exists to provide are gone. +for sym in aether_risky aether_safe; do + if ! strings "$SIZED" | grep -q "$sym"; then + echo " [FAIL] size_mode: $sym missing from the --size module" + exit 1 + fi +done + +echo " [PASS] size_mode: wasm ${BASE_SZ} -> ${SIZED_SZ} bytes, exports and symbols intact" diff --git a/tools/ae.c b/tools/ae.c index ea766402..718df624 100644 --- a/tools/ae.c +++ b/tools/ae.c @@ -382,6 +382,31 @@ static bool g_coverage = false; // with aetherc and hand-compiling it. static bool g_profile = false; +// --size: -Oz plus strip-all and dead-code elimination, for a shipped +// artifact where bytes matter more than debuggability. +// +// The other three modes are all debug-oriented -- --quick is -O0 -g, +// --profile is -O2 -g -fno-omit-frame-pointer, --coverage is -O0 -g +// --coverage -- and the default -O2 sits in the middle. Nothing pointed +// the other way, so anyone shipping a library had to emit the C and +// hand-compile it, which is exactly the hand-rolled script this is meant +// to delete. +// +// It matters most on the cross path. `zig cc` emits DWARF by DEFAULT, even +// at -O2, and nothing in the cross backend passes -g0 -- so a +// cross-compiled --emit=lib artifact is overwhelmingly debug information. +// Measured on a two-function wasi library: 956,573 bytes, of which 97.4% +// is .debug*/name sections; code and data are the rest. The equivalent +// native .so has zero .debug* sections, so this is a cross-path problem +// rather than something every target suffers. +// +// Deliberately NOT the default. A 38x size difference is discoverable -- +// anyone shipping to a browser will find the flag -- whereas stripping +// every build by default would make the first "why can't I get a stack +// trace from my wasm module" report genuinely hard to diagnose. Named +// modes keep the trade-off visible at the point of choosing it. +static bool g_size = false; + // Build an aetherc command string with optional --lib flag void build_aetherc_cmd(char* cmd, size_t cmd_size, const char* input, const char* output) { const char* emit_flag = ""; @@ -1832,6 +1857,8 @@ static bool ensure_gcc_windows(void) { // Get cflags from aether.toml [build] section (applied only for release/ae-build) // Returns empty string if not found or no aether.toml +bool ae_build_size_mode(void) { return g_size; } + const char* get_cflags(void) { static char flags[512] = ""; static bool checked = false; @@ -2276,6 +2303,17 @@ static const char* opt_flags(bool optimize) { * after coverage because --coverage's -O0 is a correctness * requirement for gcov, not a preference. */ if (g_profile) return "-O2 -g -fno-omit-frame-pointer -Wformat"; + /* --size optimises for bytes: -Os over -O2, and -g0 to suppress debug + * info the compiler would otherwise emit. Checked after --profile + * because asking for both is contradictory and the debug-oriented mode + * is the safer reading of the intent. + * + * -Os, NOT -Oz, on the native path: gcc only gained -Oz in GCC 12, and + * the CI baseline (ubuntu-22.04) ships GCC 11, where it is a hard error. + * -Os is supported by every gcc and clang we target and gives nearly the + * same result. The CROSS path can and does use -Oz, because zig bundles + * its own clang and the version is not the host's to vary. */ + if (g_size) return "-Os -g0 -Wformat"; return optimize ? "-O2 -Wformat" : "-O0 -g -Wformat"; } @@ -2497,7 +2535,7 @@ void build_gcc_cmd(char* cmd, size_t size, // #line) never run on a normal `ae build` — regressing #1252. Keep this in // sync with opt_flags(); user cflags still append after, so -Wno-format // remains an opt-out. - const char* base_opt = (g_coverage || g_profile) + const char* base_opt = (g_coverage || g_profile || g_size) ? opt_flags(optimize) : (optimize ? "-O2 -pipe -Wformat" : "-O0 -g -pipe -Wformat"); const char* trace_def = g_trace ? " -DAETHER_TRACE" : ""; @@ -2514,12 +2552,32 @@ void build_gcc_cmd(char* cmd, size_t size, const char* harden_pie = (g_emit_exe && !g_emit_lib && !g_emit_obj && !g_emit_csrc) ? " -fPIE -pie" : ""; #endif + /* --size link flags. --strip-all drops the symbol table and any debug + * sections that survived compilation; --gc-sections drops what nothing + * reaches. Both are link-time, so they apply to the runtime and stdlib + * objects too -- which is where the bulk of a library artifact comes + * from. Not applied to --emit=obj or --emit=csrc: neither links, and + * stripping an object file would remove the symbols whoever links it + * next needs. */ + /* Apple's ld is not GNU ld: --strip-all and --gc-sections are + * "unknown options" there. The equivalents are -x (strip local symbols; + * -S would also drop debug info, which -g0 already prevents) and + * -dead_strip. Same platform split harden_ldflags already makes. */ +#if defined(__APPLE__) + const char* size_link_flags = " -Wl,-x -Wl,-dead_strip"; +#else + const char* size_link_flags = " -Wl,--strip-all -Wl,--gc-sections"; +#endif + const char* size_link = (g_size && !g_emit_obj && !g_emit_csrc) + ? size_link_flags : ""; if (user_cflags[0]) - snprintf(opt, sizeof(opt), "%s%s%s%s%s %s%s", emit_lib_flags, base_opt, - harden_cflags(optimize), harden_link, harden_pie, user_cflags, trace_def); + snprintf(opt, sizeof(opt), "%s%s%s%s%s%s %s%s", emit_lib_flags, base_opt, + harden_cflags(optimize), harden_link, harden_pie, size_link, + user_cflags, trace_def); else - snprintf(opt, sizeof(opt), "%s%s%s%s%s%s", emit_lib_flags, base_opt, - harden_cflags(optimize), harden_link, harden_pie, trace_def); + snprintf(opt, sizeof(opt), "%s%s%s%s%s%s%s", emit_lib_flags, base_opt, + harden_cflags(optimize), harden_link, harden_pie, size_link, + trace_def); // Append aether_config.c to the compile when building a lib so the // aether_config_* accessors are bundled into the .so. The .c file @@ -5274,6 +5332,10 @@ static int cmd_build(int argc, char** argv) { * it. See g_profile for why neither --quick nor the default * serves. */ g_profile = true; + } else if (strcmp(argv[i], "--size") == 0) { + /* -Oz plus -g0 and link-time stripping/GC: the smallest + * artifact, for shipping rather than debugging. See g_size. */ + g_size = true; } else if (strcmp(argv[i], "--trace") == 0) { /* #1333: compile message tracing into this binary. The runtime has * to be rebuilt from source for it, since a prebuilt libaether.a @@ -5538,9 +5600,12 @@ static int cmd_build(int argc, char** argv) { if (!file) { fprintf(stderr, "Error: No input file specified.\n"); - fprintf(stderr, "Usage: ae build [-o output] [--extra file.c] [--quick] [--profile] [--target=] [-D SYMBOL]\n"); + fprintf(stderr, "Usage: ae build [-o output] [--extra file.c] [--quick] [--profile] [--size] [--target=] [-D SYMBOL]\n"); fprintf(stderr, " --quick Compile with -O0 -g for faster iteration (default: -O2)\n"); fprintf(stderr, " --profile Compile with -O2 -g -fno-omit-frame-pointer (for perf/gdb)\n"); + fprintf(stderr, " --size Compile with -Oz -g0 and strip at link, for a shipped\n"); + fprintf(stderr, " artifact (biggest win on --target, where zig emits DWARF\n"); + fprintf(stderr, " by default: a wasm --emit=lib drops ~38x)\n"); fprintf(stderr, " --target Cross-compile via zig cc: wasm, aarch64-macos, x86_64-macos,\n"); fprintf(stderr, " aarch64-linux, x86_64-linux, aarch64-freebsd, x86_64-freebsd,\n"); fprintf(stderr, " x86_64-windows, aarch64-windows (-> foo.exe; self-contained)\n"); diff --git a/tools/ae_cross.c b/tools/ae_cross.c index cd6b1ff5..7283b313 100644 --- a/tools/ae_cross.c +++ b/tools/ae_cross.c @@ -525,7 +525,14 @@ static char* wasm_export_flags(const char* c_file, const char* explicit_list) { int run_cross_compile_obj(const char* c_file, const char* obj_file, bool optimize, const char* ztriple) { const char* user_cflags = get_cflags(); - const char* opt = optimize ? "-O2" : "-O0 -g"; + /* `zig cc` emits DWARF by DEFAULT, even at -O2, and nothing here used to + * pass -g0 -- so a cross artifact was overwhelmingly debug information + * (measured: 97.4% of a two-function wasi library). --size asks for the + * smallest artifact, so it takes -Oz and suppresses that debug info. */ + /* -Oz is safe here where it is not on the native path: zig bundles its + * own clang, so the version is not the host compiler's to vary. */ + const char* opt = ae_build_size_mode() ? "-Oz -g0" + : (optimize ? "-O2" : "-O0 -g"); /* Same macos workaround as the link path: zig's bundled macOS SDK stubs * do not ship the Apple-licensed CoreAudio framework headers, so @@ -622,7 +629,14 @@ int run_cross_build(const char* c_file, const char* out_file, mkdirs(objdir); const char* user_cflags = get_cflags(); - const char* opt = optimize ? "-O2" : "-O0 -g"; + /* `zig cc` emits DWARF by DEFAULT, even at -O2, and nothing here used to + * pass -g0 -- so a cross artifact was overwhelmingly debug information + * (measured: 97.4% of a two-function wasi library). --size asks for the + * smallest artifact, so it takes -Oz and suppresses that debug info. */ + /* -Oz is safe here where it is not on the native path: zig bundles its + * own clang, so the version is not the host compiler's to vary. */ + const char* opt = ae_build_size_mode() ? "-Oz -g0" + : (optimize ? "-O2" : "-O0 -g"); const char* ex = extra ? extra : ""; /* std.audio's vendored miniaudio auto-selects a backend by platform macro: * on a macos target it #includes , an APPLE FRAMEWORK @@ -961,10 +975,25 @@ int run_cross_build(const char* c_file, const char* out_file, ? "-shared -fPIC -Wl,--export-all-symbols" : "-shared -fPIC"; } + /* --size strips at link time as well as suppressing debug info + * at compile time: --strip-all drops the symbol table and any + * debug sections that survived, --gc-sections drops what nothing + * reaches. Both apply to the runtime and stdlib objects too, + * which is where the bulk of a cross artifact comes from. The + * wasm library path already passes --gc-sections of its own; a + * second copy is harmless. */ + /* Apple targets link with Apple's ld, which rejects + * --strip-all/--gc-sections as unknown options; -x and + * -dead_strip are the Mach-O equivalents. Everything else here + * (ELF, PE, wasm) goes through an LLD that takes the GNU + * spellings. */ + const char* size_link = !ae_build_size_mode() ? "" + : (is_apple ? "-Wl,-x -Wl,-dead_strip" + : "-Wl,--strip-all -Wl,--gc-sections"); w = cross_cmd_fmt(&cmd, &cmd_cap, - "%s %s %s %s %s %s %s %s \"%s\" %s \"%s/libaether.a\" %s %s -o \"%s\" -lm", + "%s %s %s %s %s %s %s %s %s \"%s\" %s \"%s/libaether.a\" %s %s -o \"%s\" -lm", cc_cmd, sysroot_flag, apple_lib_flags, - wasm_lib_flags ? wasm_lib_flags : "", elf_pe_lib_flags, + wasm_lib_flags ? wasm_lib_flags : "", elf_pe_lib_flags, size_link, opt, feature_defs, tc.include_flags, c_file, ex, objdir, crossbuild_libs, win_platform_libs, out_file) ? 1 : -1; free(wasm_lib_flags); diff --git a/tools/ae_internal.h b/tools/ae_internal.h index bc72b11b..f18bd8bd 100644 --- a/tools/ae_internal.h +++ b/tools/ae_internal.h @@ -68,6 +68,10 @@ int run_cmd_show_warnings(const char* cmd); bool path_exists(const char* path); void mkdirs(const char* path); const char* get_cflags(void); +/* True when `ae build --size` was given: the cross backend uses it to add + * -Oz -g0 and link-time stripping. `zig cc` emits DWARF by default even at + * -O2, so without -g0 a cross artifact is overwhelmingly debug info. */ +bool ae_build_size_mode(void); /* `ae checksec` (#1646): report the hardening a linked artifact carries. * Implemented in tools/ae_checksec.c. */