Skip to content

Add a built-in pager which keeps the bottom toolbar visible - #1814

Open
tleonhardt wants to merge 13 commits into
consolidated_toolbarfrom
builtin_pager
Open

tleonhardt wants to merge 13 commits into
consolidated_toolbarfrom
builtin_pager

Conversation

@tleonhardt

@tleonhardt tleonhardt commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Summary

Stacked on #1745; please review that one first. This PR's base is consolidated_toolbar, so its diff shows only the pager.

While the bottom toolbar is running, ppaged() shows its output in a built-in pager which is a layout of the toolbar's own prompt-toolkit application. The toolbar stays on screen and keeps refreshing while the user reads. An external pager has to be given the whole terminal, so it hides the toolbar for as long as it is open.

This was part of #1745 until it was split out so the two could be judged separately. #1745 alone is complete: without this PR, ppaged() uses the external pager as it does on main.

Why have a pager of our own

Measured in a pseudo-terminal on macOS and in a Linux container, comparing this pager with less:

Built-in pager External pager (less)
Toolbar visible and refreshing while paging yes no
Works where less is not installed yes no pager at all (see #1813)
Chopped lines with sideways scrolling on Windows yes more always wraps
Regular-expression search no, plain text only yes
First page, 100,000 lines 0.03 s 0.02 s
First page, 1,000,000 lines 0.17 s 0.02 s
Leaves paged text in the scrollback no (alternate screen) yes (less -X)

Navigation, search, wide characters, color, typeahead after q, resizing and Ctrl-Z/fg behave the same in both.

What is in it

  • cmd2/pager.py (new): the view. A read-only TextArea with a search toolbar and less-like keys: Space/b, d/u, j/k, g/G, /, ?, n/N, left/right for chopped lines, q/Escape/Ctrl-C to close.
  • CommandToolbar.page(): swaps the running application's layout, key bindings and full-screen mode to the pager and back, on the display's own event loop. Output which already fits above the toolbar is printed without paging.
  • use_builtin_pager attribute, defaulting to enable_bottom_toolbar. Set it to False to keep the external pager/pager_chop. It can only turn the built-in pager off: the pager shares the toolbar's display, so where no toolbar is running the external pager is used regardless.
  • Docs in os.md, prompt.md and initialization.md, and a changelog entry.

Commits

  1. Add a built-in pager which keeps the bottom toolbar visible. The exact code, tests and docs which were removed from Consolidated toolbar during command execution #1745, restored unchanged.

  2. Parse paged output a line at a time, when the line is shown. New. The pager used to parse the whole text with prompt-toolkit's ANSI parser before showing any of it, twice, and that parser makes a fragment of every character:

    Lines First page before After Memory added before After
    10,000 0.18 s 0.01 s 86 MB none
    100,000 1.9 s 0.03 s 890 MB none
    1,000,000 21 s 0.17 s 8.8 GB none
  3. Keep the pager's pages out of iTerm2's scrollback. New, from testing in iTerm2 (242d9d96). With its default settings iTerm2 saves a screen which is erased from its top-left corner to the scrollback, the alternate screen included. A prompt-toolkit renderer erases its screen from there whenever it is reset: when the pager closes, each time the terminal is resized while it is open, and when Ctrl-Z lends the terminal. Each left the pager's page, its key summary and a toolbar row in the scrollback, and growing the window brought them back into view. A plain prompt-toolkit full-screen application does the same; less does not. The renderer always erases from the first column of a row, so while the pager has its screen a VT100 output erases the row and then from its second column to the end of the screen, which iTerm2 does not save. The cells erased and the cursor afterwards are the same. In iTerm2 3.7.7, one pager session with three resizes and a Ctrl-Z left 226 paged rows, 11 key summaries and 11 toolbar rows in the scrollback before, and none after.

What I would most like reviewed

  1. _AnsiLexer and _styled_and_plain() in pager.py. Parsing one line by itself has to give what parsing everything gave. Two things make that true: a style still in effect at the start of a line is put in front of it, and everything else the parser removes from text (which could have swallowed a newline) is removed beforehand with a regular expression written to match the parser. The tests compare the two on a list of awkward inputs and on 3,000 random ones, and a one-off run of 200,000 random inputs found no difference. The one intended difference: readline's zero-width markers (\x01…\x02) are now left out instead of being passed through to the terminal.
  2. CommandToolbar.page(). It reassigns layout, key_bindings, editing_mode and full_screen on a running application and calls renderer.erase() and renderer.request_absolute_cursor_position(). Is the restore complete on every exit path, including the display stopping while the pager is open?
  3. The default. Should the built-in pager be on whenever the toolbar is, as here, or opt-in?
  4. erasing_from_the_second_column(). It replaces erase_down on the output object for as long as the pager is open, and relies on the renderer only ever erasing from the first column of a row. Is that an acceptable way to work around a terminal, or should the pager avoid Renderer.erase() instead?

Tests

  • tests/test_pager.py: the view, its keys, search, chopping, resizing, typeahead, Rich hyperlinks, and the comparisons described above.
  • tests/test_command_toolbar.py: the pager inside the running toolbar, output which fits, redirected output, EOF while paging, and the UI-call helper page() uses.
  • tests/test_pipeline_job_control.py: a real-pty test which stops the pager with Ctrl-Z under an interactive bash and resumes it with fg.

make check, make test and make docs-test pass locally on macOS.

The pager was also run in real terminals, each typed into, resized and read back, scrollback included: iTerm2 3.7.7, Terminal.app and tmux 3.8 on macOS, and tmux 3.3a and xterm 379 on Linux. One session each of paging, three resizes while open, Ctrl-Z with fg, and quitting. Only iTerm2 kept the pager's pages in its scrollback, and no longer does after 242d9d96; the other four never did, and behave the same with and without that commit.

Not covered

  • Windows by hand. The pager's tests run on the Windows CI runners, and an earlier version was tried by a colleague on Windows, but nothing here has been measured against more in a real Windows console.
  • A style left open across thousands of lines with no reset in between makes each line look back to where the style began. Rich never writes that, since it resets after every styled piece of text, so it is not optimized.

This restores what "Take the built-in pager out, to be proposed on its own"
removed from the persistent toolbar's branch, unchanged: cmd2/pager.py,
CommandToolbar.page() and its helpers, the use_builtin_pager attribute,
their tests and their documentation.

While the bottom toolbar is running, ppaged() shows its output in a pager
which is a layout of the toolbar's own prompt-toolkit application, so the
toolbar stays on screen and keeps refreshing while the user reads. An
external pager has to be given the whole terminal, and hides it.
The pager parsed the whole text with prompt-toolkit's ANSI parser before
showing any of it, and did so twice: once to decide whether the text fits
on the screen, once to build the view. That parser makes a fragment of every
character. For 100,000 lines the first page took 1.9 seconds and the process
grew by 900 MB; for a million lines, 21 seconds and 9 GB. An external pager
shows its first page in 0.02 seconds whatever the size.

Keep the text and parse a line when the lexer is asked for it, which is when
it is about to be shown. The searchable plain text is made by removing the
style sequences with a regular expression. A text with more lines than the
screen has rows cannot fit, so it is no longer measured to find that out.

The first page now takes 0.03 seconds for 100,000 lines and 0.17 for a
million, with no memory beyond the text itself.

Parsing a line by itself has to give what parsing everything gave:

- A style still in effect when a line starts is put in front of it: the
  style sequences since the text was last reset. Rich resets after every
  piece of text it styles, so in its output there are none.
- Everything else the parser takes out of the text, which could have
  swallowed a newline, is taken out beforehand the same way: other control
  sequences, escapes which begin none, and cursor-forward, which the parser
  writes as spaces.

The tests compare the two on a list of awkward texts and on 3,000 random
ones; a one-off run of 200,000 random texts found no difference. The one
intended difference is readline's zero-width markers, which the parser
passed through to the terminal and the pager now leaves out.
@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.78%. Comparing base (4c97d59) to head (b2fe083).

Additional details and impacted files
@@                  Coverage Diff                   @@
##           consolidated_toolbar    #1814    +/-   ##
======================================================
  Coverage                 99.77%   99.78%            
======================================================
  Files                        24       25     +1     
  Lines                      7014     7293   +279     
======================================================
+ Hits                       6998     7277   +279     
  Misses                       16       16            
Flag Coverage Δ
unittests 99.78% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

iTerm2, with its default settings, saves a screen which is erased from its
top-left corner to the scrollback, and does so for the alternate screen as
well. A prompt-toolkit renderer erases its screen from there whenever it is
reset: when the pager closes, each time the terminal is resized while it is
open, and when Ctrl-Z lends the terminal. Each left the pager's page, its
key summary and a toolbar row in the scrollback of the screen the pager was
started from, and growing the window brought them back into view. A plain
prompt-toolkit full-screen application does the same; less does not.

iTerm2 only does this for an erase which starts in the corner, and the
renderer always erases from the first column of a row. While the pager has
its screen, a VT100 output now erases the row, then from its second column
to the end of the screen. The cells erased and the cursor afterwards are
the same.

In iTerm2 3.7.7, one pager session with three resizes and a Ctrl-Z left 226
paged rows, 11 key summaries and 11 toolbar rows in the scrollback before
this change, and none after it.
ppaged() does not page without one, and a CI runner does not have one.
Merging main brought in the check which stops ppaged() from trying again a
pager command the shell could not run. It landed on the condition which
guards both pagers, so once the external pager had failed, as it can for a
command run outside the command loop, the built-in pager was skipped too
and the output was printed without paging.

Decide which pager will be used before that check and apply it only to the
external one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant