Skip to content

Fix stack overflow abort on deeply nested function blocks - #1191

Open
MaxWang007 wants to merge 2 commits into
JohnnyMorganz:mainfrom
MaxWang007:cursor/fix-deep-nest-stack-overflow-d08c
Open

MaxWang007 wants to merge 2 commits into
JohnnyMorganz:mainfrom
MaxWang007:cursor/fix-deep-nest-stack-overflow-d08c

Conversation

@MaxWang007

Copy link
Copy Markdown

Fixes #1189.

StyLua 2.6.0 aborts with a stack overflow (exit 134) on the CLI's default 2MiB worker thread when checking or formatting about 100 nested function bodies. The same nesting with the closing ends missing also aborts, instead of reporting a parse error. With a larger stack, valid input formats and invalid input exits 2 with error parsing:.

format_code now switches to one fixed 16MiB stack segment when the caller has less than that free. That covers the reported 100-200 level range in release builds, and the same regressions in debug builds, which use larger frames. The segment does not grow with input depth, so a few thousand nested blocks can still overflow. wasm has no portable stack switch and keeps using the host stack.

I first tried estimating nest depth from a lexer and growing per block. That under-counted functions inside a one-line Luau interpolated string and after a \z newline inside a string, and those inputs still aborted. Those cases are now regression tests, and the scanner is gone.

Tests run on f71a3659298155e5e6440d1924cde81fff39cec2 (features lua52,lua53,lua54,luau,luajit,cfxlua unless noted):

  • Before this implementation, on the previous commit: interpolated_nested_functions_format aborted (SIGABRT); z_escape_nested_functions_format segfaulted (SIGSEGV).
  • cargo test --offline --features lua52,lua53,lua54,luau,luajit,cfxlua --test test_deep_nesting -- --test-threads=1 — pass (6 tests), including 2MiB worker threads at depths 1/20/50/100, 100 nested do blocks, unterminated functions, Luau method signatures, interpolation, and \z.
  • cargo test --offline --features lua52,lua53,lua54,luau,luajit,cfxlua --test cli -- deeply_ interpolated_ z_escape -- --test-threads=1 — pass (4 tests).
  • cargo test --offline -- --test-threads=8 — pass
  • cargo test --offline --no-default-features -- --test-threads=8 — pass
  • cargo test --offline --all-features -- --test-threads=8 — pass
  • cargo clippy --offline --all-features --all-targets -- -D warnings — pass
  • cargo fmt -- --check — pass
  • cargo check --offline --target wasm32-unknown-unknown --features luau,lua52,lua53,lua54,luajit,cfxlua — pass
  • Release CLI (cargo build --release --offline --features lua52,lua53,lua54,luau,luajit,cfxlua): 100 nested function/end formats (--check is exit 1, then 0 after format); 100 unterminated Luau method signatures --check exits 2 with error parsing:; 100-level one-line interpolation --syntax luau --check exits 1 (not 134); 4×40 \z case --check exits 1 (not 139).

This change was written with assistance from an AI coding agent. I reviewed the approach, the tests, and the remaining limits before opening this PR.

full_moon's parse_stmt keeps a large frame live across each nested block, and the formatter walks the same tree recursively. CLI worker threads only have a 2MiB stack, so about 100 nested functions abort instead of formatting valid code or returning a parse error.

Grow the stack from the input's block depth before parsing, and before each formatted block.
The depth scanner under-counted functions inside Luau interpolated strings and `\z` strings, so those inputs still aborted. format_code now switches to a single 16MiB stack when the caller has less than that free.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stack overflow (abort) on ~100 levels of nested function blocks, instead of formatting or reporting an error

1 participant