Repository navigation
Horizon: forked processes write stdout a whole line at a time (builds on #1) - #2
Merged
Merged
Conversation
Stock Horizon gives each worker a pipe that its supervisor reads and relays, so lines from different workers never mix. Forked processes write straight to the stdout they inherited, which they all share, and queue:work writes each job line in three or four pieces, so pieces from different workers interleave mid-line in the container log. A write filter on STDOUT and STDERR now holds output until a newline and passes each complete line on as its own write. A pipe delivers a write of up to PIPE_BUF in one piece, so each line reaches the log whole. A partial line is held up to 4096 bytes and passed on when the stream closes. It filters the streams themselves, which are what Symfony's ConsoleOutput writes to outside Windows, and is attached in Wiring::child() so it is in place before the command writes anything.
…nly shared stdout - Only STDOUT is filtered. After an engine fatal error (a memory or time limit) PHP stops calling user filters, and a filtered stream drops everything written to it, so with STDERR filtered Laravel's report of the error was lost. Job lines go to STDOUT; STDERR is written as stock. - A partial line still held when a stream was freed without fclose() (STDOUT at exit) made the process exit with status 255 and lost the line: PHP invalidates the stream before the filter's last flush, so stream_bucket_new() threw. The filter now makes that last bucket against a temporary memory stream, which still reaches the stream being freed, and drops rather than throws if even that fails. Output written later in shutdown (shutdown functions, destructors) stays line-buffered. - A process forked after the filter was attached (a job's own pcntl_fork()) no longer repeats the part of a line its parent held: the filter drops what it inherited when its pid changes. - The filter is attached only where stdout is shared: forked Horizon processes and shared-log workers. A normal-mode queue:work worker has its stub's own stdio and keeps stock output behaviour. - One write with many lines is split with an offset instead of copying the rest of the buffer once per line. - LineBufferFilter extends php_user_filter, PHP's own class, so PreloadTest took it for a class with a framework parent and left it out of Runtime::CLASSES; a forked child then autoloaded it from whichever release was on disk. It is preloaded now, and the test counts php_user_filter as no parent. - Comments: the console kernel reroutes Symfony's command events itself once the app has booted (outside unit tests), so CommandStarting needs no opt-in; stock Horizon relays each worker's pipe but doesn't promise whole lines either. - README: the line-at-a-time stdout under "Limitations and differences from stock". - Tests, in real PHP processes: the exit status and held tail, STDERR output after a memory-limit fatal error, output from shutdown functions and destructors, a stream closed before exit, a child forked while its parent holds part of a line, and four forked writers sharing one pipe, which mangles lines without the filter; plus many lines in one write. The process tests end themselves after 30 s rather than hang CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #1 by @serpentblade: forked Horizon processes interleave their job lines in the log, and a write filter on STDOUT keeps each line whole. This branch is #1's commit, unchanged, with one commit on top that fixes edge cases found in review. Merging it with a merge commit marks #1 as merged too.
On top of #1
stream_bucket_new()threw. The last bucket is now made against a temporary memory stream, and the filter drops rather than throws if even that fails.queue:workworkers have their stub's own stdio and keep stock behaviour.pcntl_fork()) no longer repeats the half-line its parent held.LineBufferFilteris inRuntime::CLASSES;PreloadTestskipped it because it tookextends php_user_filterfor a framework parent.Tests
New tests in real PHP processes: the exit status and held tail, STDERR output after a memory-limit fatal, output from shutdown functions and destructors, a stream closed before exit, a child forked while its parent holds part of a line, and four forked writers sharing one pipe, which mangles lines without the filter. Each fails without its fix.
composer testpasses on macOS (PHP 8.4); the new tests also pass on Linux (PHP 8.4, Docker). The Docker integration suites were not run.🤖 Generated with Claude Code