Skip to content

Stage 5: use the alternate screen for the built-in pager in reserved rendering - #1815

Draft
tleonhardt wants to merge 18 commits into
reserved_row_toolbarfrom
stage5-pager-investigation
Draft

tleonhardt wants to merge 18 commits into
reserved_row_toolbarfrom
stage5-pager-investigation

Conversation

@tleonhardt

@tleonhardt tleonhardt commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Stage 5: the built-in pager, investigated and repaired

Stage 5 asked whether the built-in pager is still worth keeping now that reserved rendering exists. The spike compared it with an external pager, and the disposition, accepted by the maintainer on 2026-11-15, is keep it and repair it.

This PR is the reserved-rendering part of that repair, plus what the Windows check of it found. The rest went where it belongs:

What Where
Print ppaged() output when the pager program cannot be run #1813, to main
The pager as its own PR, split out of #1745 #1814, stacked on #1745
Parse paged output a line at a time (first page for 100,000 lines: 1.9 s to 0.03 s) #1814; already merged into this branch through reserved_row_toolbar
Enter the alternate screen in reserved rendering this PR (9993b724)
Five defects found while checking it, none specific to the pager this PR (dc84f6c6, 3f349136, fb041f6b, 8e37a0b1, 79c44604 with 6eeecab6)
Keep the pager's pages out of iTerm2's scrollback #1814 (242d9d96); merged into this branch, with a test of it on a reservation (d181692c)

The pager defect

In reserved rendering the pager was drawn on the main screen. One pager session sent no switch to the alternate screen and one switch back; legacy rendering sends one of each. The pager painted over earlier output and left a page of paged rows, its own key summary and a second toolbar row in the scrollback. This has been so since Stage 3.

Cause. A renderer enters the alternate screen as the first thing in a full-screen frame, and marks itself as there before asking the output to switch. Under reserved rendering a frame is recorded and emitted afterwards, and the recorder refused to record the switch. The frame failed to prepare, the renderer went on believing it had switched, and the next frame was drawn without it.

The fix (9993b724)

  • RecordingOutput.enter_alternate_screen() records the switch, so it is emitted in the same batch as the frame drawn after it: both reach the terminal or neither does.
  • A frame which is not emitted has not switched screens. PromptToolkitBridge puts the renderer back to believing what it did before preparing that frame, and does not keep the screen it laid out as a baseline (_forget_screen_switch()).
  • Leaving the alternate screen stays unrecordable: a renderer does that when it is reset, never inside a frame.
  • commit()'s four "is this frame still current" checks moved into _still_current() so the rollback is written once. No change in what they check.

This also covers the pager coming back from Ctrl-Z, where the renderer leaves the alternate screen to lend the terminal and has to enter it again.

Found while checking it, and fixed here

None of these is specific to the pager. Each has a test which fails without its change.

Commit Defect Fix
dc84f6c6 Windows: command output wider than the terminal lost everything past the last column in reserved rendering. Legacy rendering wrapped it. prompt-toolkit's Windows output sets a console mode without end-of-row wrap around each flush, and its own stdout proxy turns autowrap on before each write for that reason. SerializedTerminalWriter takes that proxy's place and now does the same, for that backend only. A VT backend is sent exactly what it was before.
3f349136 A managed write landing while a frame is being prepared left the renderer's baseline at the dropped frame. The next frame was compared with a screen the terminal never received and could draw nothing. prepare() puts back the screen and cursor the renderer had before it prepared the frame.
fb041f6b A managed write landing after a frame is prepared and before it is emitted put the screen back but not the cursor. The next frame moved from where the dropped frame would have left the cursor, and was drawn that many rows off. A prepared frame carries where the cursor was before it, and the write which drops the frame puts it back.
8e37a0b1 The reserved toolbar was plain text; in legacy rendering it is a reverse-video bar. So since Stage 3, and on every platform. The painter resolves styles as an application does, with the theme laid over prompt-toolkit's default style, and gives the toolbar's text the classes prompt-toolkit's own toolbar window gives it. Theme rules for bottom-toolbar and bottom-toolbar.text now reach the text as well as the padding.
79c44604, 6eeecab6 Resizing the window while a command is printing damaged its output in reserved rendering. Seen in iTerm2: a line was printed over the old toolbar and the rest of the toolbar stayed in the output, and growing the window erased a line of output. Legacy rendering has neither. Three causes. A terminal drops the scroll region when it is resized, and the display, off the main thread, learned of a resize only from a poll or its next frame: the main thread now passes the window-change signal on. reconfigure() cleared the old toolbar at the rows it had been painted on, which iTerm2 has moved: it now clears everything below the cursor's row. And when output has just moved on to the old toolbar's row, that row is cleared too.

What to review

  1. Recording the switch at all. The recorder's rule was that screen-buffer transitions are ownership boundaries taken before a frame is prepared. Entering is now the exception, on the argument that the renderer has already laid the frame out for the whole terminal and that the reservation was released before the frame was prepared (page() suspends it first). Is that argument sound, or should the switch be taken explicitly before the first frame instead? That alternative would blank the screen for the time it takes to prepare the first frame. On Windows, mintty's log of what it was sent shows one enter and one leave for each pager session.
  2. A frame which fails part-way through being emitted keeps the renderer believing it is on the alternate screen. The switch is the first operation in a frame, so it has probably happened, and the wrong belief in that direction only costs one extra switch back.
  3. The resize change clears more than it did. After a resize during a command, every row below the cursor's is erased, and the cursor's own row too when the last command output ended a line. That is right for output which goes on from the cursor. A command which had drawn below the cursor and moved back up loses those rows until it draws them again, and one whose child process left a line unfinished without cmd2 seeing it loses that line.
  4. A signal handler during commands. While a display holding a reservation runs, the main thread has a SIGWINCH handler which asks the application for a frame and calls the handler which was there before. Is anything a cmd2 application does with that signal at odds with it?
  5. The toolbar's look changes. Anyone who has been running a reserved toolbar has seen plain text; it is now a bar, as in legacy rendering. A theme can still say otherwise.

Evidence

At the current head, 6eeecab6, on macOS:

  • make check, make test (3,896 passed, 7 skipped) and make docs-test.
  • Functional harness, reserved rendering: 126 passed with 4 known limitations (two-line toolbars, which are Stage 7, and an emulator artifact).
  • Continuity and dynamic gates at 24 rows: PASS.
  • The toolbar row in a pseudo-terminal, at 8e37a0b1: 80 of 80 cells in reverse video at the prompt, during a command and at the next prompt, in both renderings. Before that commit it was 0 of 80 in reserved rendering.

On Windows, at 9993b724 plus dc84f6c6 and 3f349136: see the results comment. examples/stage5_pager_check.py was run in Windows Terminal, conhost and mintty, in reserved and in legacy rendering. The built-in pager passes the whole checklist in all six, and its first page for 100,000 lines takes 0.06 to 0.08 s.

In iTerm2 3.7.7 on macOS, in a real window driven by AppleScript, with its screen and scrollback read back and screenshots looked at:

  • The pager checklist of examples/stage5_pager_check.py, in reserved and in legacy rendering: paging, search, colors, chopped lines and sideways scrolling, wide characters, 100,000 rows (first page after 0.02 s), a resize while open, Ctrl-C, and Ctrl-Z with fg. All pass in both, apart from what the first iTerm2 finding below leaves behind. The two renderings behave the same.
  • The reserved toolbar away from the pager: it stays on the bottom row at every sample while output scrolls, no toolbar row reaches the scrollback, and output is complete and in order. Resizing at the prompt, unfinished lines, a Rich progress bar, read_input(), select(), less run from the shell, Ctrl-Z during a command, typeahead and quitting all pass. The whole window is the shell's again while the application is stopped and after it quits.
  • The toolbar is a reverse-video bar in both renderings.
  • Two things did not pass at 8e37a0b1, and both are changed since. iTerm2 kept a page of the pager in its scrollback each time the pager closed, was resized or was stopped with Ctrl-Z, in both renderings: one session with three resizes and a Ctrl-Z left 226 paged rows before 242d9d96, measured on Add a built-in pager which keeps the bottom toolbar visible #1814's branch, and none after it there or here in either rendering. And resizing while a command printed damaged the output in reserved rendering; see the table above and the last item under "Found, not fixed here".

In four more terminals, at 6eeecab6, each driven for real (typed into, resized, and its screen and scrollback read back): Terminal.app and tmux 3.8 on macOS, and tmux 3.3a and xterm 379 on Linux in a container, xterm on a virtual X display. "Before" is 8e37a0b1, the commit before the pager and resize changes.

Runs in which resizing the window while a command printed damaged its output:

Terminal 4 lines a second, before after legacy 200 lines a second, before after legacy
iTerm2 3.7.7 5 of 8 0 of 4 0 of 4 4 of 4 2 of 8 0 of 4
Terminal.app 3 of 4 0 of 4 0 of 4 4 of 4 0 of 8 0 of 4
tmux, macOS 2 of 4 0 of 4 0 of 4 4 of 4 0 of 8 0 of 4
tmux, Linux 2 of 4 0 of 4 0 of 4 4 of 4 0 of 8 0 of 4
xterm, Linux 3 of 4 0 of 4 0 of 4 4 of 4 0 of 8 0 of 4
  • So the damage was never iTerm2's alone: every terminal tried drops the scroll region when it is resized. Where it grew, the erased line of output was seen in iTerm2, tmux on Linux and xterm.
  • The whole reserved toolbar scenario (output scrolling past the toolbar, resizes during a command and at the prompt, unfinished lines, a Rich progress bar, read_input(), select(), less from the shell, Ctrl-Z during a command, typeahead, quitting) passes in all four, as it does in iTerm2.
  • The pager: none of the four ever kept its pages in the scrollback, before or after 242d9d96, so that was iTerm2's alone, and the change makes no difference to them. Paging, three resizes while open, Ctrl-Z and fg, and quitting behave the same before and after.
  • One new thing does not pass, on Linux; see "Found, not fixed here".

At 9993b724, before the four later commits, and not repeated since:

  • In tmux, which has a real alternate screen and scrollback: reserved rendering matches legacy. The pager is on the alternate screen, earlier output is intact, and no paged rows, key summary or toolbar row are left behind.
  • Continuity and dynamic gates at 12 and 40 rows as well as 24: all PASS.
  • Functional harness, legacy rendering: 130 passed.
  • Real-pty Ctrl-Z/fg in the pager passes in both modes.
  • In a Linux container: the suite passes, the tmux checks give the same result as on macOS, and the functional harness gives reserved 126 passed with the same 4 known limitations and legacy 129 passed with 1 known (the legacy flicker).

New tests for the alternate screen: the recorder records and replays the switch; a full-screen frame through the bridge emits it once, before its content; a frame retired by a managed write or one whose preparation raised leaves the renderer not switched, and the next frame switches; the pager on a reservation sends one enter before its first page and one leave after; and after the terminal is lent during paging, two of each.

Still open for the stage

The Windows check this section used to ask for is done, and the maintainer has since tested the built-in pager by hand in Windows Terminal, with examples/stage5_pager_check.py and its rows command, and it looked good.

The later commits have since been run on Windows as well, at 6eeecab6, in Windows Terminal, conhost and mintty: see the resize results comment. The resize checks from Stage 4 were repeated there in reserved rendering before and after 79c44604, and in legacy rendering. The pager checklist was run again and gives the same results as it did before fb041f6b, 8e37a0b1 and 242d9d96, apart from how far more had paged in mintty's legacy run. The clearing rule makes nothing worse on Windows. Resizing at a prompt, with text typed, in read_input(), read_secret() and select(), around a handoff and in the pager passes in all three terminals, apart from what is listed below. Resizing while a command prints 4 or 20 lines a second damaged its output in 13 of 18 runs before these changes and in 0 of 18 after them. What is left:

Not yet looked at

  • By eye, and with a mouse, on Windows. In every Windows run the keys were injected, the windows were resized and the screen was read back by program; screenshots were looked at afterwards. The maintainer's test by hand was of the pager, in Windows Terminal.
  • Starting in a window of 2 or 3 rows, and Windows Terminal Preview. Both were checked for Stage 4 and neither was repeated at this head.
  • The Win32 console without VT processing (Win32Output) was not run.

Found, not fixed here

  • In iTerm2, resizing while a command prints very fast can still leave one line printed over the old toolbar. With a command printing 200 lines a second, that happened at every resize before 79c44604 and 6eeecab6 (8 of 8) and at 2 of 16 after them; legacy rendering, 0 of 8. It did not happen in Terminal.app, tmux or xterm (0 of 16 resizes each). What is left is output iTerm2 had already taken when it was resized, before the application could know: reinstalling the region inside the signal handler was tried and measured no better.
  • On Windows, making the window shorter while a command prints fast still damages its output, as often as before. Windows has no window-change signal, so 6eeecab6 does nothing there: the display learns of a resize from prompt-toolkit's poll or from its own next frame. With a command printing 200 lines a second, a line was printed over the old toolbar in 4 of 4 runs in each of Windows Terminal, conhost and mintty, before these changes and after them; legacy rendering, 0 of 4. It was the shrink every time and never the other three directions. Dragging the window shorter a row at a time at 20 lines a second did the same at 6 of 15 steps in Windows Terminal and in mintty. An experiment outside the product, which looks at the terminal's size before every write of command output, brought 200 lines a second to 2 of 4 runs in Windows Terminal and 0 of 4 in conhost and mintty, and the drag to none. It has not been through the test suite.
  • conhost loses output when the window grows while a command prints. Dragging it taller a row at a time at 20 lines a second lost 44 of 201 lines after these changes and 54 before them; legacy rendering, none. Not seen in Windows Terminal or mintty, nor when the window grew ten rows in one step.
  • The old toolbar row stays when a window grows from 2 rows to 3 during a command which prints nothing. At 2 rows the reservation is released and the old toolbar's text is left on row 2. At 3 rows the toolbar is painted on row 3 and the old one stays above it until the next resize. Before 79c44604 it stayed through later resizes as well. reconfigure() clears below the cursor only where it has a previous geometry, and coming back from below the two-row floor it has none. Seen in Windows Terminal and mintty, and not tried on other platforms.
  • Resizing at the instant a command line is submitted loses the line, in reserved rendering. The command runs, but the line which was typed is not left on the screen, and the first line of output is printed on the prompt's row (burst> burst 0). Under tmux on Linux, where the resize followed the keys within milliseconds: 4 of 4, before these changes and after them; legacy rendering, 0 of 4. Under tmux on macOS, where it arrived a little later: 0 of 4. Before 79c44604 a cursor report was also sometimes printed there (^[[15;8R). Not investigated. On Windows it did not happen: 0 of 8 in each of Windows Terminal, conhost and mintty, before these changes, after them and in legacy rendering.
  • conhost, a resize while the pager is open. In both renderings, quitting after shrinking the window puts the prompt a few lines too high. In reserved rendering the toolbar's old row is also left behind, below the window. The cause of the second is traced: release_region() erases the old row only if a fresh measurement equals the geometry it was given, and that geometry's buffer_id includes the window's position in conhost's buffer, which moves as lines scroll. Not seen in Windows Terminal or mintty. Not fixed, and how much conhost matters is undecided.
  • conhost, a burst of resizes at a prompt near the top of the screen leaves a second toolbar row and the prompt two or three times over: 4 of 4 tries after these changes, and before them 1 of 2 with the second toolbar and 2 of 2 with the repeated prompt. It did not happen from a full screen, or with the same resizes 1.5 s apart, or in Windows Terminal or mintty.
  • In mintty the toolbar's bar shrinks to its text once output has scrolled, until the next resize repaints it, in reserved and in legacy rendering alike. The Windows pseudo console which mintty is given does not pass the scroll region on, and repaints the toolbar's row without its padding. Not from this PR.
  • Output from another thread while the pager is open is written straight onto the pager's screen in reserved rendering, where legacy rendering puts it on the main screen. A design question, not measured.
  • more under the UTF-8 code page prints nothing, or garbage, for long lines of non-ASCII text. This is from main and affects every branch, so it belongs in an issue of its own rather than here. The built-in pager shows the same lines correctly.
  • One conhost run died at the external pager's wide-character step and did not do so again, nor in two more runs of the checklist in conhost at 6eeecab6. Not explained.
  • A toolbar style set on the session rather than in the theme is not seen by the reserved toolbar, which reads cmd2's theme. Legacy rendering uses the session's style. Noticed while fixing the bar; not changed.

Before main

  • examples/stage5_pager_check.py is to be removed, like stage4_manual.py.

Stage 5 asks whether the built-in pager is still worth keeping now that
reserved rendering exists. The comparison with an external pager has been
measured on macOS and Linux in a pseudo-terminal, which cannot be done on
Windows.

examples/stage5_pager_check.py is the Windows half: an application which
pages generated text through either pager, says which toolbar rendering and
terminal classes were selected, and reports how long Rich took to render the
text, how long the built-in pager took to draw its first page, and how long
the pager was open. Its docstring lists what to record in Windows Terminal,
conhost and mintty.

Like examples/stage4_manual.py, it is for the integration branch only and
is to be removed before the work reaches main.
In reserved rendering the pager was drawn on the main screen. One pager
session sent no switch to the alternate screen and one switch back; legacy
rendering sends one of each. The pager painted over the command's earlier
output and left a page of paged rows, its key summary and a second toolbar
row in the scrollback.

A renderer enters the alternate screen as the first thing in a full-screen
frame, and marks itself as there before it asks the output to switch. Under
reserved rendering a frame is recorded first and emitted afterwards, and the
recorder refused to record the switch. The frame failed to prepare, the
renderer went on believing it had switched, and the frame prepared next was
drawn without the switch.

Record the switch with the frame which is drawn after it, so both reach the
terminal in one piece or not at all. A frame which is not emitted has not
switched screens, so the renderer is put back to believing what it did
before preparing it, and the screen it laid out is not kept as a baseline.

This also covers the pager coming back from Ctrl-Z, where the renderer has
left the alternate screen to lend the terminal and enters it again.

Leaving the alternate screen stays unrecordable: a renderer does that when
it is reset, never inside a frame.
@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.82%. Comparing base (712f353) to head (0228e7a).
⚠️ Report is 43 commits behind head on reserved_row_toolbar.

Additional details and impacted files
@@                  Coverage Diff                   @@
##           reserved_row_toolbar    #1815    +/-   ##
======================================================
  Coverage                 99.82%   99.82%            
======================================================
  Files                        35       35            
  Lines                      9059     9214   +155     
======================================================
+ Hits                       9043     9198   +155     
  Misses                       16       16            
Flag Coverage Δ
unittests 99.82% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@tleonhardt tleonhardt self-assigned this Nov 15, 2026
…Windows

prompt-toolkit's Windows10_Output sets the console mode around every flush,
and the mode it sets does not wrap at the end of a row. Its own stdout proxy
turns autowrap on before each write for that reason. SerializedTerminalWriter
takes that proxy's place in reserved rendering and did not, so everything
past the last column of a line of command output was written over the last
column and lost. Legacy rendering wrapped the same line.

Autowrap is now turned on before each write to that backend. A VT backend is
sent exactly what it was before.

Found in Windows Terminal while running examples/stage5_pager_check.py: with
a window narrower than the timing line, the line ended "its first pags".
…prepared

A managed write which lands while a frame is being prepared retires the
frame. The write puts the renderer's last screen back to the committed one,
but the renderer is still preparing and takes the frame it is preparing in
its place when it finishes. The frame prepared next was then diffed against
one the terminal never received, drew nothing where the two agreed, and left
the terminal showing the frame before.

prepare() now notes the renderer's last screen and cursor before the frame is
prepared, and puts both back when the frame is dropped for this reason.

Not changed: output written by another thread while the built-in pager is
open still goes straight onto the pager's screen in reserved rendering.
@tleonhardt

Copy link
Copy Markdown
Member Author

Stage 5 pager check on Windows, 2026-11-15

Results of examples/stage5_pager_check.py, run in Windows Terminal, conhost and mintty, once as auto and once as legacy. The built-in pager passes the whole checklist in all six. Two defects were found and fixed along the way, and three things are reported without a fix.

Environment

  • Windows 11 Pro 10.0.26300.9974, ARM64 virtual machine.
  • Windows Terminal 1.25.3131.0; conhost from the same Windows build; mintty 3.8.3 from Git for Windows, started as Git Bash starts it (login bash, MSYSTEM=MINGW64). Python gets a pseudo console there without winpty.
  • Python 3.15.2, the x86-64 build from the uv venv, so emulated on this machine. prompt-toolkit 3.0.53, Rich 15.0.0.
  • Tree: 9993b724 plus the two fixes below, since committed as dc84f6c6 and 3f349136. Windows Terminal auto was run again after merging reserved_row_toolbar (31d46236, which carries main's 0047276a) with the same results.
  • Launch: .venv\Scripts\python.exe examples\stage5_pager_check.py auto|legacy, with NO_COLOR unset.

Method

Keys were injected into the application's console with WriteConsoleInput, not typed. The screen was read with the console API; the scrollback through UI Automation in Windows Terminal, from the whole screen buffer in conhost, and from mintty's log of what it was sent. Window screenshots were taken at each step and a sample of them looked at. Scripts, results, logs and screenshots are kept outside the repository, in toolbar-flicker-harness/runs/20261115-stage5-windows-pager/.

1. info

Terminal Mode Rendering Output / input Size TERM
Windows Terminal auto reserved ReservedOutput / Win32Input 119x29 None
Windows Terminal legacy legacy Windows10_Output / Win32Input 119x30 None
conhost auto reserved ReservedOutput / Win32Input 119x29 None
conhost legacy legacy Windows10_Output / Win32Input 119x30 None
mintty auto reserved ReservedOutput / Win32Input 119x29 xterm-256color
mintty legacy legacy Windows10_Output / Win32Input 119x30 xterm-256color

The window is 120 columns in each: prompt-toolkit's Windows outputs report one fewer. The built-in pager is the default in all six, and the external pagers are 'more' / chop 'more'.

2 to 6. Built-in pager

The same in all six runs unless a row says otherwise.

Check Result
rows 2000 color brings up the built-in pager yes
Toolbar clock visible and ticking while paging yes
Space, b, G, g work
/row 150 Enter, then n work; n goes to row 1500
row N green, bold red red yes
After q: hist 0 to hist 39 intact yes
After q: paged rows, key summary or second toolbar left behind none
Alternate screen: one enter and one leave for each pager session yes (mintty's log)
rows 2000 long chop: right arrow reaches tail0, left returns yes, 4 presses each way
rows 2000 wide: wide characters wrap without breaking yes
Resize while open, page on, quit re-laid out, one toolbar; see finding 4 for conhost
Ctrl-C while open closes the pager; prompt and toolbar return

Timing lines, in seconds. "First page" is from handing the text to the pager to its first frame.

Run 2000 color: Rich / first page 100000: Rich / first page
Windows Terminal auto 0.10 / 0.02 0.52 / 0.06
Windows Terminal legacy 0.08 / 0.02 0.53 / 0.06
conhost auto 0.08 / 0.02 0.58 / 0.08
conhost legacy 0.09 / 0.02 0.52 / 0.06
mintty auto 0.08 / 0.02 0.59 / 0.06
mintty legacy 0.09 / 0.02 0.55 / 0.06

7. External pager (more)

The same in all six runs.

  • Colors are shown, not printed as escape sequences.
  • Space pages and Enter moves one line. It cannot go back or search: b and / do nothing.
  • The toolbar is hidden while it runs and returns, ticking, when it exits.
  • It pages on the main screen, so every row it showed stays in the scrollback.
  • chop has no effect: long lines wrap.
  • A resize is not followed; it goes on paging at the old height.
  • Ctrl-C ends it and the prompt returns.
  • rows 100000: Rich takes 0.52 to 0.60 s. The check measures no first-page time for it.
  • rows 2000 wide shows nothing. more exits after 0.04 s having printed a blank line or nothing. See finding 3.

Findings

1. Fixed (dc84f6c6): command output wider than the terminal was cut off in reserved rendering

In reserved rendering on Windows, everything past the last column of a line of command output was lost: each character there was written over the one before it. Legacy rendering wrapped the line. Seen first as a timing line ending its first pags, then with rows 2 long, which lost both of its tails.

prompt-toolkit's Windows10_Output sets the console mode around each flush, and the mode it sets has no wrap at the end of a row. Its own stdout proxy turns autowrap on before each write for that reason. SerializedTerminalWriter, which takes the proxy's place in reserved rendering, did not. It now does, for that backend only; a VT backend is sent exactly what it was before. Confirmed in Windows Terminal before and after. Not specific to the pager: it affects any command output in reserved rendering on Windows.

2. Fixed (3f349136): the defect this PR lists as "found, not fixed here"

A managed write which lands while a frame is being prepared left the renderer's baseline at the dropped frame. PromptToolkitBridge.prepare() now puts back the screen and cursor the renderer had before it prepared the frame. The new test fails without the change: the frame after the dropped one emitted no content.

Not changed, and a design question rather than a defect of this kind: output which another thread writes while the pager is open is written straight onto the pager's screen in reserved rendering, where legacy rendering puts it on the main screen. That was so before and is so for a write between frames as well. Not measured here.

3. more cannot show long lines of non-ASCII text under the UTF-8 code page

From main, not from this PR: cmd2 sets the console to code page 65001 while an external pager runs. Under it, piped to more in a plain cmd.exe:

  • a line of 2 wide characters is shown;
  • a line of 50 or 100 wide characters prints nothing;
  • a line of 200 é prints part of it and then garbage from memory;
  • a line of 200 ASCII characters is shown.

Under code page 437 the same lines are shown complete, as mojibake. The built-in pager shows all of them correctly.

4. conhost: a resize while the pager is open

conhost keeps a 9001-line buffer and a window onto it. Two things follow, neither seen in Windows Terminal or mintty.

  • Both modes. After shrinking the window while the pager is open, quitting puts the prompt a few lines too high. Two or three earlier lines of the main screen are overwritten, and the lines which were below them are left under the bottom of the window.
  • Reserved only. The toolbar's old row is left there as well, as a second toolbar row below the window. The old row is in fact left on the main screen whenever the pager opens, resize or not: release_region() erases it only if a fresh measurement equals the geometry it was given, and the geometry's buffer_id holds the window's position in the buffer, which conhost moves with every line scrolled. Traced: given (0, 0, 120, 9001), measured (0, 14, 120, 9001), not erased. Without a resize the toolbar is repainted on the same row when the pager closes, so nothing shows. A shell command and quitting the application leave no stale row.

This is the "backing buffer taller than the viewport" case which the Stage 1 Windows probes never exercised.

5. The reserved toolbar is drawn without its bar

At the prompt, a reserved toolbar is plain text. In legacy rendering, and in the pager in either mode, it is a reverse-video bar. The painter is given DynamicStyle(get_pt_theme), cmd2's theme alone, which has no rule for bottom-toolbar; prompt-toolkit merges its default UI style into an application's, and that is where reverse comes from. Not specific to Windows. Not changed: it may be intended.

6. One run died

The first conhost auto run ended at the external pager's rows 2000 wide step: the application was gone and the script driving it with it. A second complete run, and separate attempts at that step, did not repeat it. Not explained.

Limits

  • Keys were injected, not typed, and nothing was judged by a person at the terminal; the screenshots were looked at afterwards.
  • Timings are from an emulated x86-64 Python in a virtual machine.
  • mintty's scrollback was judged from what mintty was sent and from the visible screen.
  • The legacy Win32 console without VT processing (Win32Output) was not run.
  • iTerm2 is still to be checked by eye on macOS.

…e it is emitted

A managed write which lands after a frame is prepared, and before it is
emitted, drops the frame. The write puts the renderer's last screen back to
the committed one, but the renderer's cursor stayed where the dropped frame
would have left it. The frame prepared next moved from there, and was drawn
that many rows from where it belonged: one row too high where the dropped
frame had left the cursor a row lower.

A prepared frame now carries where the renderer's cursor was before it was
prepared, and the write which drops the frame puts it back. This is the
other half of 3f34913, which did the same for a write landing while the
frame is still being prepared.
The reserved toolbar was plain text. In legacy rendering it is a
reverse-video bar across the whole row.

Two things differed. The painter resolved styles against cmd2's theme alone,
which has no rule for bottom-toolbar; an application lays its style over
prompt-toolkit's default one, and "reverse" comes from there. And the
toolbar's text had no class at all: prompt-toolkit's window gives every cell
class:bottom-toolbar and its text class:bottom-toolbar.text besides, so a
rule for either reached only the padding after the text.

The painter is now given the default style with the theme over it, and a
style to put under every fragment's own. A theme's rules still win.

Found on Windows while running examples/stage5_pager_check.py, and not
specific to it. In a pseudo-terminal on macOS the bottom row went from 0 to
80 of 80 cells in reverse video, at the prompt and during a command, as it
is in legacy rendering.
… corner

There the erase reaches the terminal through the reserved adapter, or
replayed from a recorded frame, and the adapter is what the application
holds. Without looking through it to the backend, no erase was changed.
…oved it

After a resize the old band's text is still on the screen, and where depends
on the terminal. reconfigure() cleared the rows it had been painted on, and
only when the terminal had grown, which is right for a terminal that adds
rows at the bottom. iTerm2 brings rows back from its scrollback instead, so
the old band moves down with everything else and the rows it was painted on
hold output: a line of output was erased each time the window grew. Made
narrower, iTerm2 rewraps the band's row into two, and one was left.

The rows of the old band are below the cursor's row wherever the terminal
has put them, and nothing else is: output goes on from the cursor, and a
prompt redraws itself after a resize. reconfigure() now makes room for the
new band and then clears everything below the cursor's row, after any
resize.

One case is not below it. A terminal sets its scroll region back to the
whole screen when it is resized, so the next line of output to end moves the
cursor on to the old band's row instead of scrolling. The line after it was
then written over the old toolbar and the rest of the toolbar stayed in the
output. The bridge now notes when the last thing to move the cursor was
command output ending a line, which leaves it at the start of a row holding
none of that output, and that row is then cleared as well. It is erased from
its second column, with the first cleared separately, because iTerm2 saves a
screen erased from its top-left corner to the scrollback.

In iTerm2 3.7.7, resizing while a command printed 200 lines a second left a
line printed over the toolbar at every resize before this and the next
commit (8 of 8), and at 2 of 16 after them. Legacy rendering: 0 of 8.
A terminal sets its scroll region back to the whole screen when it is
resized. Until the display has installed the region again, command output
runs on to the toolbar's row. The display runs off the main thread, where
prompt-toolkit cannot attach the window-change signal and looks at the
terminal's size twice a second instead; the bridge also looks whenever a
frame is drawn, which is as often as the toolbar refreshes. A command
printing four lines a second had its output damaged at most resizes.

The signal comes to the main thread, where the command is. While a display
holding a reservation runs, a handler there asks the application for a
frame, which is where the bridge looks at the size. It calls whatever
handler was installed before it, puts that one back when the display stops,
and is not installed off the main thread or where there is no such signal.

Reinstalling the region inside the handler instead, in the thread doing the
writing, was tried and measured no better: what is left is output the
terminal had already taken when it was resized.

In iTerm2 3.7.7, with a command printing four lines a second, resizes in
four directions damaged the output in 5 runs of 8 before this and the
previous commit and in 0 of 4 after them.
@tleonhardt

Copy link
Copy Markdown
Member Author

Resize checks on Windows after 79c44604 and 6eeecab6, 2026-11-15

The Stage 4 resize checks, repeated on Windows at 6eeecab6, in Windows Terminal, conhost and mintty. Each was run in reserved rendering at the head, in reserved rendering at the tree from before the two resize commits, and in legacy rendering.

The clearing rule makes nothing worse on Windows and several things better. The resize checks at a prompt pass in all three terminals, apart from one conhost case which was there before (finding 4). Resizing while a command prints is clean at 4 and at 20 lines a second, where it was not before. What is left is damage when the window is made shorter during fast output, which is as common after the change as before it: Windows has no window-change signal, so 6eeecab6, which closed that gap elsewhere, does nothing here.

Environment

  • Windows 11 Pro 10.0.26300.9974, ARM64 virtual machine.
  • Windows Terminal 1.25.3131.0; conhost from the same Windows build; mintty 3.8.3 from Git for Windows, started as Git Bash starts it.
  • Python 3.15.2, the x86-64 build from the uv venv, so emulated. prompt-toolkit 3.0.53.
  • "After" is the branch head, 6eeecab6. "Before" is d181692c, the commit before 79c44604. It differs from the 8e37a0b1 the PR's table uses only by the pager's scrollback change and tests.
  • The test suite on this machine at the head (python -Xutf8 -m pytest tests): 3,704 passed, 199 skipped. PR CI: 30 of 30.

Method

A check application outside the repository (resize_app.py: prompt check> , toolbar TOOLBAR HH:MM:SS, refresh interval 0.5 s) was run in a real window of each terminal. A driver set the window to exact rows and columns by moving its edges, injected keys with WriteConsoleInput, and judged each step three ways:

  • the console's screen buffer, characters and attributes, read in one call so that rows agree;
  • a screenshot of the window, searched for bright full-width bands, which is what the toolbar's bar looks like;
  • in Windows Terminal, the terminal's own text through UI Automation, scrollback included. In conhost the whole 9001-line buffer. mintty offers neither, so there the screen was sampled about 30 times a second while output ran, which sees every line before it scrolls away.

Nothing was dragged with a mouse or watched by a person. Scripts, logs, samples and screenshots are kept outside the repository, in toolbar-flicker-harness/runs/20261115-stage5-windows-resize/.

Resizing at a prompt and around handoffs: all pass

The same in Windows Terminal, conhost and mintty at the head, unless a finding below says otherwise.

Check Result
Idle prompt, 30 rows to 2, 3, 4, 30, 3, 2, 3, 4, 30; from an empty screen and from a full one one toolbar on the bottom row at every size, one prompt, no key needed; output above intact
The same with text typed and not submitted, then 90 and 100 columns text survives every resize and the command then runs
Sixteen resizes one straight after another, from a full screen settles with one toolbar, one prompt, output intact
A drag: a row at a time down to 12 rows and back, 0.1 s apart the same
Resize at the instant a line is submitted, 8 times the typed line stays above its output, 8 of 8 (also before, and in legacy)
A quiet command and a line left unfinished, through 4, 3, 2, 3, 4, 30 rows and 90 columns PARTIALEND whole on one row; toolbar right except at one step, finding 3
read_input(), read_secret(), select(): shorter, taller, narrower, wider toolbar on the bottom row each time; the answer is accepted
suspend_bottom_toolbar() twice nested, resized while the guest has the terminal no toolbar while the guest runs; back on the bottom row of the new size afterwards
3, 2, 3, 30 rows after a shell command returned right at each size
Built-in pager open: shorter, taller, narrower, wider, page on, quit toolbar on the bottom row and key summary above it at each size; no paged rows or key summary left
Toolbar lines in everything the terminal holds, after all of it one (Windows Terminal, conhost)
Quit, then 60 lines from the shell the shell has the whole window; no toolbar row, no bar

At 2 rows an idle prompt shows the fallback toolbar on row 2, and at 3 the reserved toolbar is back on row 3, as on 2026-11-14.

Resizing while a command prints

Runs in which the output was damaged. Each run resizes four ways: shorter (30 to 20 rows), taller, narrower (120 to 90 columns), wider.

Terminal 4 lines a second, before after legacy 200 lines a second, before after legacy
Windows Terminal 1.25 1 of 4 0 of 4 0 of 4 4 of 4 4 of 4 0 of 4
conhost 3 of 4 0 of 4 0 of 4 4 of 4 4 of 4 0 of 4
mintty 3.8.3 3 of 4 0 of 4 0 of 4 4 of 4 4 of 4 0 of 4

At 20 lines a second: 2 of 2 before, 0 of 2 after and 0 of 2 in legacy, in each terminal.

  • The damage is the one the PR describes: a line printed over the old toolbar, with the rest of the toolbar left in the output (out 030418:35:06).
  • At 200 lines a second it is the shrink every time, and only the shrink: 12 of 12 shrinks damaged before and 12 of 12 after, against 0 of 36 of the other three directions in both.
  • A drag at 20 lines a second, 15 steps shorter and 15 taller a row at a time, 0.2 s apart:
Terminal Before After Legacy
Windows Terminal 6 lines printed over the toolbar, all while shrinking 6, the same none
mintty 6, all while shrinking 6, the same none
conhost 5 printed over the toolbar and 54 of 201 lines lost, while growing 1 and 44 lost none

Findings

1. Not fixed on Windows: shrinking the window during fast output

On POSIX the PR measures 0 of 8 at 200 lines a second after the change in every terminal but iTerm2. On Windows it is 4 of 4 in all three terminals, before and after. The application learns of a resize from prompt-toolkit's poll or from its own next frame, and by then more than one line has been written; clearing the row output has just moved on to covers only the first.

An experiment, not product code: the check application was run with PromptToolkitBridge.before_managed_write patched to call _reconfigure_if_resized() first, so that the size is looked at before every write of command output.

Terminal 200 lines a second, after with the experiment Drag at 20 lines a second, after with the experiment
Windows Terminal 4 of 4 runs 2 of 4 6 lines over the toolbar none
conhost 4 of 4 0 of 4 1 over the toolbar, 44 lost none
mintty 4 of 4 0 of 4 6 over the toolbar none

The other steps it was run through (resize at submit, quiet, unfinished line, 1, 4 and 20 lines a second) gave the same results as without it, with one exception: in Windows Terminal, with a line left unfinished, the toolbar was not repainted at 3 rows after 2, where without the experiment it was. One try each. It costs one console call for each write and was not run through the test suite.

2. conhost loses lines of output when the window grows during output

In the drag, each step taller lost the lines written until the application caught up: out 0094 is followed by out 0103. 54 of 201 lines before the change and 44 after; none in legacy. Not seen in Windows Terminal or mintty, and not when the window grew by ten rows in one step. With the experiment of finding 1: none.

Also conhost only, and the same before and after: at one line a second through 2, 3, 4 and 30 rows, one line was printed over the toolbar (out 000418:38:28).

3. The old toolbar row stays when a quiet command's window grows from 2 rows to 3

In Windows Terminal and mintty, during a command which prints nothing: at 2 rows the reservation is released and the old toolbar's text stays on row 2, no longer updated. At 3 rows the toolbar is painted on row 3 and the old one is still on row 2, for as long as the size stays at 3 rows.

  • Before the change it stayed longer: still there at 4 rows, and with an unfinished line at 30 rows too. After it, the next resize clears it. So the clearing rule helps here.
  • TerminalDisplay.reconfigure() clears below the cursor only where it has a previous geometry. Coming back from below the two-row floor it has none, and only makes room.
  • Not seen in conhost, where nothing is left on screen at 2 rows.

4. conhost: a burst of resizes at a prompt near the top of the screen

Fourteen or more resizes with no pause, starting from a prompt near the top of a mostly empty screen, left a second toolbar row and the prompt two or three times over. At the head 4 of 4 tries; before the change 1 of 2 with the second toolbar and 2 of 2 with the repeated prompt. The same sizes 1.5 s apart were right in both, and from a full screen, which is what the checklist's burst step uses, it did not happen in either. Not seen in Windows Terminal or mintty.

5. mintty: the bar shrinks to its text once output scrolls

8e37a0b1 on Windows: the reserved toolbar is a reverse-video bar across the window in all three terminals. In mintty, after any command output has scrolled, only the toolbar's text is in reverse video and the rest of the row is black, until the next resize repaints it. That was so in 9 of the 10 looks taken after a command's output; the tenth followed a resize with no scrolling after it. Legacy rendering is the same, 9 of 10, so the reserved toolbar now matches legacy there too.

mintty's log of what it was sent shows why. The Windows pseudo console does not pass the scroll region on: for each line of output it scrolls the whole screen, blanks the row the toolbar moved to, and repaints the toolbar as its text followed by a cursor move over the padding. Not from this PR, and not seen in Windows Terminal, whose pseudo console is its own.

The pager checklist at the head

examples/stage5_pager_check.py was run again at 6eeecab6 with the driver of the earlier run, in the same six combinations, for fb041f6b, 8e37a0b1 and 242d9d96, which that run predates.

  • Windows Terminal, conhost and mintty in reserved rendering, and Windows Terminal and conhost in legacy: every recorded value is the same as in the earlier run, clock times and timings aside. That covers paging, search, colors, chopped lines, wide characters, 100,000 rows, a resize while open, Ctrl-C, and what is left behind after quitting. The one error is the one known from that run: more shows nothing for rows 2000 wide.
  • mintty in legacy rendering differs in the external pager only: more was one page further back at each step (row 30 at the top after Space, where it was row 60). The built-in pager's values are the same. Not looked into; neither legacy rendering nor more is touched by these commits.
  • First page of 100,000 rows: 0.06 to 0.07 s, as before.

Limits

  • Keys were injected and windows resized by program. Nothing was dragged with a mouse or judged by a person at the terminal; a sample of the screenshots was looked at afterwards.
  • Small numbers: four runs at 4 and at 200 lines a second, two at 20, and one drag, for each terminal and tree. The experiment of finding 1 is one pass.
  • mintty's scrollback cannot be read, so "output above intact" and the count of toolbar lines held are from Windows Terminal and conhost only. In mintty the damage counts come from sampling the pseudo console's screen, which showed the same damaged rows that Windows Terminal's own text did.
  • Legacy runs were judged by the terminal's own text where there is one: the sampling reads the legacy toolbar being erased and redrawn as damage.
  • Starting the application in a window of 2 or 3 rows was not repeated, nor Windows Terminal Preview, nor the Win32 console without VT processing (Win32Output).
  • Python is an emulated x86-64 build in a virtual machine, which stretches every race measured here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant