Skip to content

Consolidated toolbar during command execution - #1745

Open
tleonhardt wants to merge 55 commits into
mainfrom
consolidated_toolbar
Open

tleonhardt wants to merge 55 commits into
mainfrom
consolidated_toolbar

Conversation

@tleonhardt

@tleonhardt tleonhardt commented Sep 5, 2026 •

Copy link
Copy Markdown
Member

Summary

Today enable_bottom_toolbar=True only shows a toolbar while prompt-toolkit is waiting for input — it vanishes the moment a command starts running. This branch makes the toolbar persistent: it stays visible and keeps refreshing while commands execute, with command output scrolling above it.

Because a persistent toolbar owns the bottom of the terminal, anything else that wants the terminal has to coordinate with it. That drives the other piece of this branch: a suspension mechanism for the places that genuinely need the raw terminal (nested prompts, shell commands, Python/IPython shells, pagers, and pipes to interactive programs).

The built-in pager is no longer part of this PR. It kept the toolbar visible while paging, and it was about 300 of this PR's 1,050 source lines and a third of its tests. It is now proposed separately in #1814, stacked on this PR, so that the two can be judged separately. Here, ppaged() uses the configured external pager exactly as it does on main, and the toolbar is hidden while the pager has the terminal, which #1743 says is acceptable.

What changed

Persistent toolbar (cmd2/command_toolbar.py, new)

  • CommandToolbar reuses main_session.app and its existing toolbar container rather than standing up a second prompt-toolkit application, so content callback, styling, and refresh interval are shared with the prompt.
  • Commands still run on the main thread; the command loop drives the toolbar UI from a background thread.
  • Stable ToolbarStream wrappers route Python-level and piped subprocess output through prompt-toolkit's output proxy so writes land above the toolbar. Their identities stay consistent across cmd2 redirection and toolbar suspension.
  • An RLock shared by the toolbar and its streams serializes writes against the proxy swap during pause/resume.
  • Input stays coordinated: cursor position reports are handled, type-ahead is saved for the next prompt, Ctrl-C is forwarded to the process group (os.killpg on POSIX, _thread.interrupt_main() on Windows), and Ctrl-Z suspend-to-background is honored where supported.
  • If the display thread dies on its own, _app_exited() detaches the streams so output goes back to the real terminal and the error is reported, instead of writing into a dead proxy.

Suspension (cmd2/cmd2.py)

  • New public Cmd.suspend_bottom_toolbar() context manager for application code that calls input(), drives its own terminal UI, or starts a subprocess that inherits the terminal.
  • Applied internally via a @suspend_toolbar decorator on _read_raw_input(), select(), _run_shell_command() (which runs both shell and the editor), _run_python(), do_ipy(), and _run_cmdfinalization_hooks(), and with a with block around the external pager in ppaged().
  • read_input() and read_secret() show the same toolbar in their own prompt, refreshing at the main prompt's refresh_interval, so it stays visible while a command waits for input. select() does not, because prompt-toolkit's choice() cannot refresh one.
  • Pipes to interactive targets: the pipe process now inherits the real terminal through the new pipe_target() helper, and the toolbar suspends for the life of the pipe (held in RedirectionSavedState.pipe_context, released in _restore_output()). Streams with no file descriptor still fall back to PIPE. Previously alias | fzf-style commands produced no output at all.
  • The toolbar is started around onecmd_plus_hooks() from the interactive loop and around startup commands. It is not started for non-interactive input or for onecmd_plus_hooks() calls made outside the loop.
  • A failing toolbar.stop() no longer disables the toolbar for the rest of the session — _command_toolbar = None moved into its own finally.

Notes and trade-offs

  • get_bottom_toolbar() now runs in a background UI thread during command execution. Applications that read state a command mutates should guard it with a lock. This is documented.
  • One guarded dependency on PromptSession's internal layout structure, verified against prompt-toolkit 3.0.52 and 3.0.53; it raises a clear RuntimeError if the structure moves.
  • Ctrl-\ (SIGQUIT) is inert while the toolbar is up because raw mode clears ISIG. This matches existing behavior at the main prompt, so it is documented rather than changed.

Tests

  • New tests/test_command_toolbar.py (72 tests) covering startup/shutdown, output routing, unfinished lines, suspension paths, pipe handling, Ctrl-C and Ctrl-Z, display failures, and the EOF shutdown path.
  • tests/test_pipeline_job_control.py gains a real-pty test which suspends a running command with Ctrl-Z under an interactive bash and resumes it with fg. On Linux it fails with the fix below disabled.
  • Includes a fix for Windows CI: the fixture now binds the app session to the test pipe and recording output, so patch_stdout() cannot lazily construct a real Win32Output and raise NoConsoleScreenBufferError on runners with no console screen buffer.

Docs

  • docs/features/prompt.md: persistent behavior, refresh, suspend_bottom_toolbar(), and threading guidance.
  • docs/features/os.md: the toolbar is hidden while a pager is open.
  • examples/getting_started.py: new work [seconds] command demonstrating the toolbar clock updating while a command prints output.
  • CHANGELOG entry under 4.5.0.

Hardening after the first round (2026-11-14)

Found by driving the branch in a pseudo-terminal on macOS and Linux, each fixed in its own commit with a regression test:

  • Editor ran inside the live toolbar. 4.4.2 made run_editor() call _run_shell_command() directly, bypassing the suspension on do_shell(). The suspension now sits on _run_shell_command().
  • Ctrl-Z then fg left the terminal cooked on Linux. After resuming, keys were held until Enter and cursor reports were echoed. The display thread now waits for the stop to take it before it touches the terminal again.
  • A raising get_bottom_toolbar() flooded the terminal. Each traceback asyncio logged triggered a redraw, which raised again: 229 reports during one 1.5 second command. The display now stops on its first error, reports it once, and the command carries on without a toolbar. A display that cannot restart after a pause is reported rather than raised into the command or out of cmdloop().
  • Unfinished lines of output were erased. Working... followed later by finished displayed as finished, a completed \r progress line vanished, and a final line without a newline was lost intermittently. This is upstream patch_stdout behavior for text flushed without a newline. The proxy now ends an unfinished line on screen so the toolbar starts on a row of its own, and continues the line when more output arrives. Verified with wrapped lines, wide characters, color carried across writes, and a Rich progress bar. One visible effect: a line still unfinished when a command ends is ended there, so the next prompt starts on a new line.

Windows check (2026-11-14)

Found by driving the branch in real terminal windows on Windows 11 (Windows Terminal, conhost, mintty and the legacy console), each fixed in its own commit with a regression test:

  • An unfinished line which filled the last column wrote over the line above it. prompt-toolkit's Windows outputs report one column fewer than the console wraps at, so the rows of such a line were miscounted. It took a line exactly as wide as the window, or a column or two short of a multiple of that: the 200-column line of the manual check failed in a window 100 columns wide and passed at the default 120. Rows are now counted by the width of the console's screen buffer.
  • An unfinished line which reached the last row showed its first row twice. Ending the line had already scrolled that row off the screen, and a toolbar of several rows made it happen a row sooner for each extra row. A line is now continued only while it and the toolbar fit on the screen together; otherwise it stays ended. This one is not specific to Windows. Checking it on macOS and Linux found the toolbar's rows were not yet known for a command's first output, where prompt-toolkit leaves the toolbar out until the terminal has reported its cursor position. A toolbar of two rows was allowed for as one, so a line one row too tall for it was still continued. The rows the toolbar last took are now remembered from one frame to the next, starting with the main prompt's (2f4b5dd).
  • On a legacy console, an unfinished line which exactly filled its rows showed its last row twice. A legacy console ("Use legacy console", where prompt-toolkit uses Win32Output) moves to the next row as soon as a row is full, where other terminals wait for the next character. The first fix above introduced this: counting by the size prompt-toolkit reports had happened to be right for these lines there. One more row is now counted for such a line on that output.

On a legacy console the manual check's color command prints its escape sequences as text, with or without the toolbar. That is the console, not this branch.

Known limitations

  • The toolbar is erased and redrawn around each prompt and each write. That flicker predates this branch and is not addressed here; reserved_row_toolbar is the follow-up for it.
  • Lock held across a pause. If get_bottom_toolbar() takes a lock, a command must not hold that lock while it prompts, pages, or runs a subprocess: pausing the toolbar waits for the callback to return, so the application hangs. Documented with a warning in docs/features/prompt.md. Bounding that wait is implemented on reserved_row_toolbar and comes with it.

NOTES

This is one possible architecture and approach to solving this problem.

I did a bunch of manual testing on both macOS and Linux and everything looks good there, but I need someone else to help test on Windows.

TODO

  • Improve documentation
  • Investigate slow tests on Windows - they take about 7 times as long on Windows relative to Linux. Before this change Windows tests only took about 40% longer than Linux. The culprit appears to be slow Windows tests in the test_command_toolbar.py file.
  • Investigate stuck test on Windows when specifically using Python 3.11. One of the tests in test_command_toolbar.py gets stuck indefinitely
  • Manual testing on Windows - got a colleague to verify everything behaves as expected using the getting_started.py app on Windows from this branch
  • Manual check on Windows of output which stops in the middle of a line, in Windows Terminal, conhost, mintty and the legacy console. Results and the three fixes which followed are in the comments

Closes #1743

tleonhardt and others added 7 commits September 5, 2026 17:25
… and refreshing during command execution. Prompts, pagers, and shell commands temporarily suspend it while using the terminal.

Try the work command examples/getting_started.py for a demonstration.

The feature uses two prompt-toolkit displays at different times: the existing PromptSession while waiting for input, and a dedicated CommandToolbar while commands execute.

- Command execution stays on the main thread. The command loop starts the toolbar’s UI in a background thread, using the same content callback, styling, and refresh interval.
- Output appears above the toolbar. Stable stream wrappers route Python and piped subprocess output through prompt-toolkit’s output proxy. Their identities remain consistent across cmd2 redirection and toolbar suspension.
- Input remains coordinated. The toolbar handles terminal position reports, saves typed-ahead keys for the next prompt, and forwards Ctrl-C to the main thread.
- Other terminal interfaces get exclusive access. Nested prompts, pagers, and shell commands temporarily suspend the toolbar. Custom commands can do the same with suspend_bottom_toolbar().
- Cleanup restores normal terminal operation. When execution ends, buffered output is flushed, workers are stopped, and the original streams are restored.

Most supporting code lives in cmd2/command_toolbar.py, with lifecycle integration in cmd2/cmd2.py. Because get_bottom_toolbar() runs in the UI thread during commands, shared mutable state should be protected with a lock.
1. Pipes to interactive targets (cmd2.py:3366, command_toolbar.py:43) — the pipe process now inherits the real terminal via the new pipe_target() helper, and the toolbar suspends for the life of the pipe (held in RedirectionSavedState.toolbar_suspension, released in _restore_output). Streams with no file descriptor still fall back to PIPE. Verified under a real pty: alias | python3 -c "...isatty()" printed PIPE_ISATTY True; before the fix that command produced no output at all.

2. Ctrl-C not reaching subprocesses (command_toolbar.py:161) — POSIX now uses os.killpg(os.getpgrp(), SIGINT), matching what the terminal driver does. Verified under a pty with a child that reports its own SIGINT: post-fix CHILD_GOT_SIGINT appears and the child exits immediately; pre-fix it never fired and the child ran to completion. Windows keeps interrupt_main() — broadcasting a console control event there would hit unrelated processes on the same console. Ctrl-\ stays inert, which matches the main prompt's existing behavior; documented rather than changed.

3. Failing stop() disabling the toolbar (cmd2.py:2098) — _command_toolbar = None moved into its own finally.

4. Writes racing _pause() (command_toolbar.py) — an RLock shared by the toolbar and its ToolbarStreams serializes writes against the proxy swap. The lock is deliberately released before _thread.join(), since the toolbar thread can itself be blocked writing through a stream.

5. Post-startup failures swallowed — new _app_exited() detaches the streams when the display stops on its own, so output goes to the terminal instead of a dead proxy, and reports the error. Worth noting: the review's stated trigger doesn't hold — a get_bottom_toolbar() that raises on refresh is caught by asyncio's default handler and the thread survives. The fix targets the case where the thread genuinely dies.

6. Final flush cancelled before writing — does not reproduce. Application.run_async already awaits wait_for_cpr_responses() and any in-flight run_in_terminal futures before its loop closes (application.py:770-779). I built the exact scenario — an output with responds_to_cpr=True and a CPR future confirmed pending at pause time — and the trailing text arrived with and without the proposed fix. I reverted my drain implementation, which was adding a run_in_terminal round-trip and up to a second of CPR waiting at every command boundary for nothing. I kept the test as a regression guard and a two-line guard against Application.exit() raising in a loop callback if the app has already stopped.

7. Docs/example mismatch — do_work now uses Annotated[int, Argument(nargs="?")], so work and work 5 both work. Verified in the real app.
Three tests called _read_raw_input() after the toolbar context had exited,
so no prompt-toolkit app session was active. patch_stdout() then asked the
default app session for its output, which lazily builds a real terminal
Output: harmless Vt100_Output on POSIX, but Win32Output on Windows, which
raises NoConsoleScreenBufferError without a console screen buffer of the
kind GitHub Actions does not provide.

Bind the fixture's app session to the test pipe and recording output so
every prompt-toolkit call in these tests resolves to the test terminal
rather than the ambient one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C7nkx8kKC2mpaMGfLwXe6J
…d an embedded pager

Following prompt-toolkit’s [layout composition model](https://python-prompt-toolkit.readthedocs.io/en/3.0.52/pages/full_screen_apps.html#the-layout).

- CommandToolbar (cmd2/command_toolbar.py:101) now reuses main_session.app and its actual toolbar container, eliminating the second application and duplicated display configuration.

- The new pager (cmd2/pager.py:37) shares that application. The toolbar continues refreshing during scrolling, searching, and resizing. Colors, wrapped lines, and horizontal scrolling are supported.

- Enabling the toolbar selects the embedded pager. Set self.use_builtin_pager = False to retain your external pager.

An external pager such as less takes control of the terminal; a layout alone cannot reserve space around it. That path still temporarily suspends the toolbar.

This simplifies rendering ownership, although the embedded pager adds code.

The command UI thread and output proxy remain necessary for synchronous commands. There’s also one guarded dependency on PromptSession’s internal layout structure, verified against versions 3.0.52 and 3.0.53.
Closing the terminal's input ends the display's Application.run() with
EOFError, which _resume() swallows because it is an ordinary shutdown
rather than a failure the running command should hear about. Nothing
exercised that handler, leaving it as the one uncovered line in the new
module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C7nkx8kKC2mpaMGfLwXe6J
@codecov

codecov Bot commented Sep 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.77%. Comparing base (70217f0) to head (4c97d59).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1745      +/-   ##
==========================================
+ Coverage   99.75%   99.77%   +0.01%     
==========================================
  Files          23       24       +1     
  Lines        6665     7014     +349     
==========================================
+ Hits         6649     6998     +349     
  Misses         16       16              
Flag Coverage Δ
unittests 99.77% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

page() constructed a full Pager -- a TextArea, a SearchToolbar, an ANSI
lexer and its key bindings -- and then threw it away whenever the output
fit on screen, which is a common case for ppaged(). Only the line
measurements were ever used on that path.

Move the size check into a module-level output_fits() that measures the
text directly, and build the Pager only once the output has to be
scrolled. A shared _fragments() helper does the one ANSI parse for both,
so the measurement cannot drift from what the view would render.

The fast path's size check drops from ~0.15ms to ~0.02ms per call, so it isn't a massive speedup or anything, but still cleaner.
1. Chop-mode paging no longer loses your column — cmd2/pager.py:_scroll()
The destination column now comes from the window's current horizontal_scroll when chop is set, instead of target * window_width (always 0 in chop mode)

2. A toolbar that can't start no longer takes the session down - cmd2/cmd2.py
_command_toolbar_context() catches startup failure, reports once via perror, sets a new _command_toolbar_disabled flag so later commands don't retry and re-spam and runs the command without a toolbar

3. ppaged() no longer starts a toolbar outside the command loop — it now pages inside a toolbar only when one is already running, and otherwise falls through to the external pager. The documentation has been updated to contain truthful statements in this regard.

4. use_builtin_pager documented as opt-out only — initialization.md, os.md, prompt.md and the ppaged() docstring now state that it cannot turn the embedded pager on where no toolbar is running, since the two share one display.
…g visible hyperlink text, colors, and width calculations

When ppaged() receives Rich output containing hyperlinks, Rich emits OSC 8 sequences that prompt-toolkit's ANSI parser does not understand. Their contents become visible pager text: a linked click here renders as 8;id=...;https://example.comclick here8;;. This also corrupts width calculations and searchable text.

This change strips unsupported OSC sequences before passing captured output to cmd2's new internal pager.
…toolbar.py

Also:
- Moved common code used by test_command_toolbar.py and test_pager.py to conftest.py

Since the pager is effectively its own component, it feels cleaner to have the tests in their own file.
@tleonhardt tleonhardt self-assigned this Sep 6, 2026
tleonhardt and others added 7 commits September 6, 2026 00:10
Cause: rich's Style.render() skips the OSC 8 hyperlink wrapper entirely when legacy_windows=True. Console(force_terminal=True) on Windows auto-detects a legacy console, so the fixture string never contained \x1b]8; — the failure was in the test's setup, not in the pager.

Fix (tests/test_pager.py:31): pass legacy_windows=False to the Console, so the fixture emits real hyperlink sequences on every platform.
test_command_toolbar_ui_call_reports_display_failure patched
call_soon_threadsafe on the toolbar's live event loop and swallowed every
call, not just the one it meant to drop. The patch was still installed
while asyncio.run() tore the loop down, and asyncio resolves the default
executor's join future through call_soon_threadsafe, so that report was
dropped too.

On Linux and macOS prompt_toolkit registers stdin with loop.add_reader, no
default executor is ever created, and shutdown_default_executor() returns
immediately. On Windows prompt_toolkit parks its WaitForMultipleObjects
watcher in loop.run_in_executor(), so the executor exists and the loop
waits on a future nothing resolves: 300 seconds on Python 3.12 and later,
and forever before that, where shutdown_default_executor() takes no
timeout. Windows CI went from 34s to 345s on 3.14, and hung for six hours
on 3.11.

Drop only the request made on the command thread and pass everything else
through. The display still dies with the ValueError the test asserts on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej3Nyn6vGngxYqm9FA1Jw5
@tleonhardt
tleonhardt marked this pull request as ready for review September 6, 2026 15:13
…Windows

It now has three commands, in the order your colleague should try them:

1. slow — sleeps 5 seconds. Does TOOLBAR IS HERE stay visible the whole time? This is the discriminating test: if the toolbar isn't up during a command, the real bug is that CommandToolbar never starts on Windows, and the pager is just collateral damage.

2. longout — pages 500 lines. Built-in pager ends in q: quit; the external one shows -- More --.

3. diag — dumps the guard state to real stderr (so it survives whatever the pager does to the screen).

The single most useful thing to report is the slow result plus the diag block — that alone should help narrow down the underlying root cause.
prompt-toolkit draws the toolbar starting at the cursor, and erases it from the
first column of that row before printing anything more above it. Output which
stopped in the middle of a line was erased along with the toolbar: a line
written as 'Working...' and finished later lost its first part, a progress line
redrawn with carriage returns vanished once it was complete, and a last line
without a newline was lost whenever the display redrew before it stopped.

End an unfinished line on the screen so the toolbar always starts on a row of
its own, and when more output arrives move back up and write the unfinished
line again before it, which puts the cursor and any style it set where they
were. A line unfinished when the display stops stays ended, so the next prompt
starts on a new line, as it does with reserved rendering.
Stopping the display after an error draws a last frame and fires after_render
again. A handler which fails there replaced the error already recorded, so the
report named a consequence instead of the cause.

That is how a race in the pager navigation test surfaced on CI. Its
after_render check read the terminal's size after another thread had resized
it, so a frame drawn at the old size was judged against the new one. asyncio
used to log and ignore that assertion. Now it stops the display, and the check
failed again on the last frame with an unrelated AttributeError. Judge each
frame against the size it was drawn at.
@tleonhardt

Copy link
Copy Markdown
Member Author

Manual check wanted on Windows

One fix on this branch (fb4a2d9, keeping lines of output which stop before a newline) has been verified in real terminals on macOS and Linux, but on Windows only by CI. It moves the cursor through prompt-toolkit's output backend, so it is worth five minutes in a real console.

Save this next to a checkout of consolidated_toolbar as toolbar_check.py:

"""Manual check for cmd2 PR #1745: command output which stops in the middle of a line.

Run from a checkout of the consolidated_toolbar branch:

    uv run python toolbar_check.py

Then type each command below at the prompt and compare with what its help says.
A clock in the bottom toolbar should keep ticking the whole time.
"""

import sys
import time

import cmd2


class Check(cmd2.Cmd):
    def __init__(self) -> None:
        super().__init__(allow_cli_args=False, enable_bottom_toolbar=True, refresh_interval=0.5)
        self.prompt = "check> "

    def get_bottom_toolbar(self) -> str:
        return f"TOOLBAR {time.strftime('%H:%M:%S')}"

    def do_working(self, _: cmd2.Statement) -> None:
        """Expect 'Working...' for two seconds, then the SAME line reads 'Working... done'."""
        self.stdout.write("Working...")
        self.stdout.flush()
        time.sleep(2)
        self.stdout.write(" done\n")

    def do_progress(self, _: cmd2.Statement) -> None:
        """Expect one line counting 0% to 100% in place, which stays as 'progress 100%'."""
        for percent in range(0, 101, 10):
            self.stdout.write(f"\rprogress {percent:3d}%")
            self.stdout.flush()
            time.sleep(0.3)
        self.stdout.write("\n")
        self.poutput("after progress")

    def do_wrap(self, _: cmd2.Statement) -> None:
        """Expect a run of x wider than the window, then 'END' directly after the last x."""
        self.stdout.write("x" * 200)
        self.stdout.flush()
        time.sleep(2)
        self.stdout.write("END\n")

    def do_color(self, _: cmd2.Statement) -> None:
        """Expect 'red and still red' on one line, all of it red."""
        self.stdout.write("\x1b[31mred")
        self.stdout.flush()
        time.sleep(2)
        self.stdout.write(" and still red\x1b[0m\n")

    def do_last(self, _: cmd2.Statement) -> None:
        """Expect 'first line', then 'last line, no newline', then the prompt on a NEW line."""
        self.poutput("first line")
        self.poutput("last line, no newline", end="")

    def do_lines(self, _: cmd2.Statement) -> None:
        """Expect five numbered lines, one per half second, and nothing else."""
        for number in range(1, 6):
            self.poutput(f"line {number}")
            time.sleep(0.5)


if __name__ == "__main__":
    sys.exit(Check().cmdloop())

Run it with uv run python toolbar_check.py, then type each command and compare with the table. help <command> prints the same expectation.

Command What you should see
working Working... for two seconds, then that same line reads Working... done
progress one line counting 0% to 100% in place, left as progress 100%, with after progress below it
wrap a run of x wider than the window, then END directly after the last x
color red and still red on one line, all of it red
last first line, last line, no newline, then the prompt on a new line
lines five numbered lines, one per half second, nothing else

Throughout, the clock in the bottom toolbar should keep ticking and stay on the bottom row. The toolbar blinking as output is printed is expected on this branch.

What would be a failure: the first half of a line disappearing ( done without Working...), the progress 100% line vanishing, END on a line of its own or overwriting the xs, escape sequences printed as text such as ←[1A, or the toolbar left behind in the middle of the output.

Please try it in Windows Terminal, and in the legacy console (conhost.exe) or mintty if you have them, and report for each:

  • Windows build and terminal program with its version
  • Python version
  • which commands, if any, did not match

@tleonhardt
tleonhardt marked this pull request as ready for review November 14, 2026 22:25
@tleonhardt

tleonhardt commented Nov 14, 2026 •

Copy link
Copy Markdown
Member Author

Ready for review

@kmvanbrunt @bambu

This is now the single PR for #1743 (#1744 is closed; everything in it is here). It makes enable_bottom_toolbar=True keep the toolbar visible and refreshing while commands run, with no API change. It does not remove the redraw flicker that has always been there; that is separate follow-up work on reserved_row_toolbar.

The source change is about 1,000 lines in four files. Suggested reading order, with what I would most like a second pair of eyes on:

  1. cmd2/command_toolbar.py, CommandToolbar (start with start(), _resume(), _pause(), stop()). A prompt-toolkit application runs on a background thread while the command stays on the main thread. Please check the pause/resume ordering and the locking: the RLock shared with ToolbarStream, and the rule that the lock is released before the display thread is joined.
  2. _ContextStdoutProxy._write_and_flush() in the same file. This is the newest code and the part I am least sure reviewers will like. prompt-toolkit erases from column 0 of the row the toolbar starts on, which destroyed output that stopped before a newline (Working..., \r progress lines). The proxy now ends such a line on screen and continues it when more output arrives. It overrides a private upstream method. The alternative is to accept upstream's behavior and document it.
  3. CommandToolbar.page() and cmd2/pager.py. The embedded pager swaps layout and key bindings on the live application and restores them. Is use_builtin_pager defaulting to enable_bottom_toolbar the right default, or should the embedded pager be opt-in?
  4. cmd2/cmd2.py: _command_toolbar_context(), suspend_bottom_toolbar(), and the eight places the toolbar is suspended (six @suspend_toolbar decorators and two with self.suspend_bottom_toolbar() blocks). The question is whether anything that takes over the terminal was missed. One was, until 4.4.2 exposed it (run_editor()), so a fresh look is worth it.

Coupling to prompt-toolkit internals, all in command_toolbar.py: the shape of PromptSession's layout (guarded, fails with a clear error), StdoutProxy._start_write_thread() and _write_and_flush(), get_typeahead()/store_typeahead(), and a few renderer calls for the pager's full-screen switch. The requirement is prompt-toolkit>=3.0.53, the only version tested.

How it has been tested:

  • CI: Linux, macOS and Windows on 3.11 through 3.15, including free-threaded builds.
  • 91 unit tests for the toolbar and pager, plus a real-pty test that suspends and resumes the pager under an interactive bash.
  • Driven in a pseudo-terminal on macOS and Linux: output ordering, typeahead, nested prompts, pager, editor, shell commands, pipes, redirection, Ctrl-C, Ctrl-Z, resize, multiline toolbars, unfinished lines, and failing toolbar callbacks.
  • Not yet done: the manual Windows check in the comment above.

Known limitations are listed in the description. The one worth an opinion: a command that holds a lock which get_bottom_toolbar() also takes, across a prompt, pager or subprocess, hangs the application. It is documented with a warning rather than fixed here.

To try it: uv run examples/getting_started.py, then work 5, help -v, shell ls, and edit.

@tleonhardt

Copy link
Copy Markdown
Member Author

Windows manual check: results

Replying to the request for a manual check on Windows.

One failure: wrap goes wrong when the unfinished line fills the last column of the window. At the default 120-column window all six commands match in every terminal, so a check at the default size alone passes.

How it was run: a script opened real terminal windows, typed the commands into the console, sampled the screen about every 30 ms, and took screenshots of the windows, which were then inspected. It was not watched by eye in real time.

Environment

  • Windows 11 Pro 26H2, build 26300.9974, ARM64
  • Python 3.14.10 x64 (emulated), prompt-toolkit 3.0.53, branch at 809a1d5

Results

Each terminal got two passes of the six commands: once near the top of the window, and once with output scrolling at the bottom.

Terminal Width Commands that did not match
Windows Terminal 1.25.3131.0 120 none
Windows Terminal Preview 1.26.3133.0 120 none
conhost.exe (inbox) 120 none
mintty 3.8.3 (Git Bash) 100 (its size on this machine) wrap
mintty 3.8.3 120 none (only working and wrap run)
Windows Terminal 1.25 and conhost.exe 100 wrap (only working and wrap run)

Throughout, the toolbar clock ticked and stayed on the bottom row, and no escape sequences appeared as text. conhost's scrollback had no toolbar left behind in it.

The failure

After the two-second pause in wrap, the check> wrap line is overwritten by xs, and END lands on the start of a row of xs instead of after the last one:

check> working
Working... done
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   <- was "check> wrap"
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ENDxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
check>
  • Cause: on Windows, prompt-toolkit's get_size() reports one column fewer than the console wraps at (Win32Output.get_size(), which Windows10_Output uses too). _ContextStdoutProxy._rows() divides by that number, so it moves the cursor up one row too many.
  • When it happens: the unfinished line is exactly as wide as the window, or within one column short of two widths, two short of three, and so on. For the 200 xs that means windows 200, 100, 67, 50 or 40 columns wide. It depends on the width, not on the terminal program.
  • Also at the default size: an extra command which writes a line exactly as wide as the window fails the same way at 120 columns in both conhost and Windows Terminal.

Fix

A fix with a regression test is written but not pushed yet. _rows() now counts rows by the width of the console's screen buffer, which is where the console wraps, when the output is one of prompt-toolkit's Windows outputs.

With the fix, checked in the same real terminals:

  • wrap and the exact-width line are right at 100 columns in conhost, Windows Terminal and mintty.
  • A conhost whose screen buffer (150) is wider than its window (120) is right too, for lines of 120, 130 and 200 columns.
  • The six commands at 120 columns, two passes each in conhost and Windows Terminal, give the same output as before the fix.

Not covered

The "Use legacy console" mode, where prompt-toolkit falls back to Win32Output and no escape sequences are used, was not tested. It needs a global registry setting changed.

prompt-toolkit's Windows outputs report one column fewer than the console
has, to keep what it draws out of the last column. Raw command output is
wrapped by the console itself, at the full width of its screen buffer. Rows
were counted by the reported size, so an unfinished line which filled the
last column was counted as one row longer than it was: continuing it went up
one row too many, wrote over the line above, and left the rest of the output
on top of the line's own last row.

That takes a line exactly as wide as the window, or within one column short
of two widths, two short of three, and so on. The 200 columns of the manual
check's wrap command fail in a window 100 columns wide and pass in one of
120, which is why the default window size showed nothing.

Use the width of the console's screen buffer when the output has one. That
is also right for a console whose buffer is wider than its window, which
wraps at the buffer's width.
An unfinished line was continued whenever its width fitted in the rows and
columns of the screen. A line which reaches the last row has already lost
its first row by then: ending it scrolls the screen, so going back up stops
one row short of where the line began, and writing the line again from there
shows its first row twice.

A toolbar of several rows scrolls the screen sooner, by a row for each row
it has beyond the first, so a bound which allows for one row is not enough.

Continue a line only while it and the toolbar below it fit on the screen
together. The display tells the proxy how many rows its toolbar took when it
was last drawn. A line too long for that stays ended, as one longer than the
screen already did, and the output which follows starts on the next row.
@tleonhardt

Copy link
Copy Markdown
Member Author

Follow-up: both fixes are pushed

Following the Windows check results above, two fixes are now on the branch, each with regression tests.

ac0cc0b: count the rows of an unfinished line by the width Windows wraps at. This is the wrap failure reported above. _rows() now uses the width of the console's screen buffer when the output is one of prompt-toolkit's Windows outputs.

cbfb566: leave an unfinished line ended once its first row has scrolled away. This one was found during the same check and is not in the report above. An unfinished line which reached the last row of the window had its first row shown twice when it was continued, because ending the line had already scrolled that row off. A toolbar of several rows made it happen a row earlier for each extra row. A line is now continued only while it and the toolbar fit on the screen together; otherwise it stays ended and the output which follows starts on the next row. The logic is not specific to Windows, but it has only been run on Windows.

Checked in real terminals on the same machine as before, with both fixes:

  • wrap and a line exactly as wide as the window are right at 100 columns in conhost, Windows Terminal and mintty.
  • A conhost whose screen buffer (150) is wider than its window (120) is right for lines of 120, 130 and 200 columns.
  • In conhost at 120x30, with toolbars of one, two and three rows, unfinished lines stopping zero to three rows short of the bottom show every row exactly once.
  • The six commands of the manual check at 120 columns, two passes each in conhost and Windows Terminal, give the same output as before the fixes.

Still not covered: the "Use legacy console" mode.

@tleonhardt

tleonhardt commented Nov 15, 2026 •

Copy link
Copy Markdown
Member Author

Note on the legacy console

The "Use legacy console" mode, listed as not covered in the Windows check results, could not be tested on my Windows VM. The legacy console is not installed on it (Windows 11 26H2 ARM64, no ConhostV1.dll): with ForceV2 set to 0, conhost.exe exits at once with 0x80070057.

As the closest substitute, prompt-toolkit was forced onto its Win32Output backend and non-VT input reader, which is what a legacy console would get, inside a normal conhost.exe at 120 columns. With both fixes:

  • All six commands of the manual check matched.
  • Unfinished lines one column short of, exactly at, and one column past one and two window widths were continued correctly, with nothing above them written over.
  • The toolbar stayed on the bottom row with its clock ticking, and no escape sequences appeared as text.

This exercises prompt-toolkit's cursor movement through the Win32 API, but the console underneath is still the modern one, so it is not a full stand-in. A real legacy console prints the raw escape sequences of the color command as text, and may differ for a line which exactly fills a row. That still needs a machine with the legacy console installed.

A terminal which understands escape sequences leaves the cursor in the last
column when a row has just been filled, and wraps at the next character. A
legacy Windows console moves to the next row at once. Ending an unfinished
line which exactly fills its rows therefore leaves the cursor a row lower
there. Rows were counted as for a terminal which waits, so continuing such a
line went up one row too few and wrote it again a row below itself, which
showed its last row twice.

Counting rows by the size prompt-toolkit reports, one column short, happened
to give the right answer for these lines on a legacy console. Counting by the
width the console wraps at fixed every other console and broke this one.

A legacy console gets prompt-toolkit's Win32Output, which unlike its other
Windows outputs has no vt100 output. Count one more row for a line which has
just filled a row when the output is that one.
@tleonhardt

Copy link
Copy Markdown
Member Author

Legacy console: tested, one more fix pushed

The legacy console is now installed on the test machine and the manual check has been run in it. This replaces the earlier note, which could only test a stand-in.

Correction to that note: it cited conhost.exe exiting with 0x80070057 as the sign that the legacy console was missing. That was the wrong evidence. Starting conhost.exe with a command fails that way in legacy mode even when the legacy console is installed. It really was missing at the time (the Microsoft.Windows.Console.Legacy capability was NotPresent), and it is installed now.

Setup

  • Windows 11 Pro 26H2, build 26300.9974, ARM64, with the Microsoft.Windows.Console.Legacy~~~~0.0.1.0 capability installed and HKCU\Console\ForceV2 set to 0 for the test
  • Console 120 columns by 30 rows, Python 3.14.10 x64, prompt-toolkit 3.0.53
  • The app reported output=Win32Output input_reader=ConsoleInputReader

Results with the two earlier fixes

  • Five of the six commands matched. color printed ←[31mred and still red←[0m on one line. It prints exactly the same with the toolbar disabled, so that is the legacy console showing the check script's raw escape sequences, not the toolbar.
  • The toolbar stayed on the bottom row with its clock ticking, and nothing was left behind in the scrollback.
  • One failure: an unfinished line exactly one or two window widths wide had its last row shown twice. The same lines were right with the toolbar disabled.

Cause and fix

A legacy console moves the cursor to the next row as soon as a row is full, where a terminal which understands escape sequences waits for the next character. Ending a line which exactly fills its rows therefore leaves the cursor a row lower there, and continuing it went up one row too few.

This was introduced by ac0cc0b. Counting rows by the size prompt-toolkit reports, one column short, happened to give the right answer for these lines on a legacy console.

939ce21 counts one more row for a line which has just filled a row when the output is prompt-toolkit's Win32Output, the one a legacy console gets. It comes with a regression test.

Checked with all three fixes

  • Legacy console: lines of exactly one, two and three window widths, and one column either side of one and two, are continued correctly. The six commands are unchanged, two passes.
  • conhost.exe and Windows Terminal 1.25, not in legacy mode: the same exact-width lines and wrap, working and progress are still right.

An unfinished line is continued only while it and the toolbar fit on the
screen together, and the display reported the toolbar's rows from the frame
it last drew. Each run of the application forgets what it drew, and
prompt-toolkit leaves the toolbar out until the terminal has reported where
its cursor is. A command which writes at once does so before that report
arrives, so the toolbar counted for no rows and a toolbar of two rows was
allowed for as one: a line one row too tall for it was continued, and its
first row was shown twice.

Windows knows the height of its console without asking, so this showed only
on terminals which answer a cursor position request.

Keep the rows from the last frame which drew the toolbar, starting with the
main prompt's, which are the rows it is about to take again.
@tleonhardt

Copy link
Copy Markdown
Member Author

The same checks on macOS and Linux: one more fix

cbfb566 (leave an unfinished line ended once its first row has scrolled away) had only been run on Windows, so I ran it in a pseudo-terminal on macOS and in a Linux container. Each case writes an unfinished line of numbered rows about as tall as the screen, finishes it, and checks that every row is shown exactly once, counting the rows which scrolled off the top.

One failure: with a two-row toolbar on a 24-row screen, a 23-row line still showed its first row twice.

Cause: the display reported the toolbar's rows from the frame it last drew. Each run of the prompt-toolkit application forgets what it drew, and on a terminal which answers cursor position requests the toolbar is left out until the answer arrives. A command which writes at once does so before that, so the toolbar counted for no rows and was allowed for as one. Windows knows the height of its console without asking, which is why this did not show there.

Fix: 2f4b5dd. The rows from the last frame which drew the toolbar are kept, starting with the main prompt's, with a regression test.

Checked at 2f4b5dd, which includes the legacy console fix 939ce21:

Check macOS Linux
Lines of 20 to 30 rows, one-row and two-row toolbars, fresh and scrolled screens (28 cases) all right all right
The other pseudo-terminal scenarios (100 checks) pass pass
Test suite 3,072 pass 3,071 pass

The same 28 cases against 809a1d5, before the Windows fixes, fail in 6, so the check does detect what cbfb566 fixed.

Not rerun on Windows: 2f4b5dd changes nothing there unless a frame is drawn without the toolbar, but it has not been run in a real Windows console.

The fix which holds the display thread until a Ctrl-Z stop has taken it was
only tested through the built-in pager, which was the convenient way to have
a command waiting. The fault is in the command display, whatever the command
is doing.

Share the test's body and add a command which keeps printing. On Linux with
the fix disabled both fail, with the display's cursor reports echoed after
fg; with it both pass.
The persistent toolbar and the pager which keeps it visible while paging
arrived together, but they are two decisions. The pager is about 300 of this
branch's 1,050 source lines and a third of its tests, and issue #1743 itself
allows the toolbar to disappear "when using a pager".

Remove cmd2/pager.py, CommandToolbar.page() and the helpers only it used,
the use_builtin_pager attribute, their tests and their documentation.
ppaged() goes back to the configured external pager, with the toolbar
suspended while it has the terminal, which is the path use_builtin_pager =
False already took.

The pager returns in a pull request of its own, stacked on this one.
@tleonhardt

Copy link
Copy Markdown
Member Author

The built-in pager has moved to its own PR

To make this easier to review, I have taken the built-in pager out of this PR and proposed it separately in #1814, which is stacked on this one.

What changed here (ead92b0):

  • cmd2/pager.py, CommandToolbar.page() and the helpers only it used, and the use_builtin_pager attribute are gone, with their tests and docs.
  • ppaged() uses the configured external pager exactly as it does on main. The toolbar is hidden while the pager has the terminal and comes back when it exits, which Bottom bar should persist during command execution #1743 says is acceptable.
  • The source change is now about 700 lines, down from 1,050, and the tests about 1,300, down from 1,840.

One test was added first (7747214): the real-pty test for Ctrl-Z used the pager as its waiting command, so it now also suspends an ordinary command. On Linux, with the Ctrl-Z fix disabled, it fails; with the fix it passes.

In the reviewer guide above, point 3 no longer applies here. It asked about CommandToolbar.page(), pager.py and the use_builtin_pager default; those questions are now in #1814. Points 1, 2 and 4 stand.

Checked after the change: make check, make test (3,038 passed) and make docs-test; CI is 32/32; and the pseudo-terminal scenarios on macOS (125 pass), including that the toolbar is hidden while less is open and back on the bottom row afterwards. The Windows manual check earlier in this thread was about unfinished lines of output and is not affected.

Related: #1813 makes ppaged() print its output when the pager program is not installed. It is independent of this PR and matters more without a built-in pager.

# Conflicts:
#	CHANGELOG.md
#	docs/features/prompt.md
…toolbar

The example and its documentation came from main, where the bottom toolbar
is only on the screen while the prompt is. They said so, and that cmd2 does
not keep a toolbar there while a command runs. On this branch it does.

Say what the example still offers: a toolbar drawn outside the terminal the
cmd2 application runs in, which therefore stays when something takes over
that terminal, as a shell command does, where cmd2 has to hide its own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bottom bar should persist during command execution

1 participant