Repository navigation
Add a built-in pager which keeps the bottom toolbar visible - #1814
Open
tleonhardt wants to merge 13 commits into
Open
tleonhardt wants to merge 13 commits into
tleonhardt wants to merge 13 commits into
Conversation
This restores what "Take the built-in pager out, to be proposed on its own" removed from the persistent toolbar's branch, unchanged: cmd2/pager.py, CommandToolbar.page() and its helpers, the use_builtin_pager attribute, their tests and their documentation. While the bottom toolbar is running, ppaged() shows its output in a pager which is a layout of the toolbar's own prompt-toolkit application, so the toolbar stays on screen and keeps refreshing while the user reads. An external pager has to be given the whole terminal, and hides it.
The pager parsed the whole text with prompt-toolkit's ANSI parser before showing any of it, and did so twice: once to decide whether the text fits on the screen, once to build the view. That parser makes a fragment of every character. For 100,000 lines the first page took 1.9 seconds and the process grew by 900 MB; for a million lines, 21 seconds and 9 GB. An external pager shows its first page in 0.02 seconds whatever the size. Keep the text and parse a line when the lexer is asked for it, which is when it is about to be shown. The searchable plain text is made by removing the style sequences with a regular expression. A text with more lines than the screen has rows cannot fit, so it is no longer measured to find that out. The first page now takes 0.03 seconds for 100,000 lines and 0.17 for a million, with no memory beyond the text itself. Parsing a line by itself has to give what parsing everything gave: - A style still in effect when a line starts is put in front of it: the style sequences since the text was last reset. Rich resets after every piece of text it styles, so in its output there are none. - Everything else the parser takes out of the text, which could have swallowed a newline, is taken out beforehand the same way: other control sequences, escapes which begin none, and cursor-forward, which the parser writes as spaces. The tests compare the two on a list of awkward texts and on 3,000 random ones; a one-off run of 200,000 random texts found no difference. The one intended difference is readline's zero-width markers, which the parser passed through to the terminal and the pager now leaves out.
5 tasks done
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## consolidated_toolbar #1814 +/- ##
======================================================
Coverage 99.77% 99.78%
======================================================
Files 24 25 +1
Lines 7014 7293 +279
======================================================
+ Hits 6998 7277 +279
Misses 16 16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
iTerm2, with its default settings, saves a screen which is erased from its top-left corner to the scrollback, and does so for the alternate screen as well. A prompt-toolkit renderer erases its screen from there whenever it is reset: when the pager closes, each time the terminal is resized while it is open, and when Ctrl-Z lends the terminal. Each left the pager's page, its key summary and a toolbar row in the scrollback of the screen the pager was started from, and growing the window brought them back into view. A plain prompt-toolkit full-screen application does the same; less does not. iTerm2 only does this for an erase which starts in the corner, and the renderer always erases from the first column of a row. While the pager has its screen, a VT100 output now erases the row, then from its second column to the end of the screen. The cells erased and the cursor afterwards are the same. In iTerm2 3.7.7, one pager session with three resizes and a Ctrl-Z left 226 paged rows, 11 key summaries and 11 toolbar rows in the scrollback before this change, and none after it.
ppaged() does not page without one, and a CI runner does not have one.
Merging main brought in the check which stops ppaged() from trying again a pager command the shell could not run. It landed on the condition which guards both pagers, so once the external pager had failed, as it can for a command run outside the command loop, the built-in pager was skipped too and the output was printed without paging. Decide which pager will be used before that check and apply it only to the external one.
# Conflicts: # CHANGELOG.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #1745; please review that one first. This PR's base is
consolidated_toolbar, so its diff shows only the pager.While the bottom toolbar is running,
ppaged()shows its output in a built-in pager which is a layout of the toolbar's own prompt-toolkit application. The toolbar stays on screen and keeps refreshing while the user reads. An external pager has to be given the whole terminal, so it hides the toolbar for as long as it is open.This was part of #1745 until it was split out so the two could be judged separately. #1745 alone is complete: without this PR,
ppaged()uses the external pager as it does onmain.Why have a pager of our own
Measured in a pseudo-terminal on macOS and in a Linux container, comparing this pager with
less:less)lessis not installedmorealways wrapsless -X)Navigation, search, wide characters, color, typeahead after
q, resizing and Ctrl-Z/fgbehave the same in both.What is in it
cmd2/pager.py(new): the view. A read-onlyTextAreawith a search toolbar andless-like keys: Space/b,d/u,j/k,g/G,/,?,n/N, left/right for chopped lines,q/Escape/Ctrl-C to close.CommandToolbar.page(): swaps the running application's layout, key bindings and full-screen mode to the pager and back, on the display's own event loop. Output which already fits above the toolbar is printed without paging.use_builtin_pagerattribute, defaulting toenable_bottom_toolbar. Set it toFalseto keep the externalpager/pager_chop. It can only turn the built-in pager off: the pager shares the toolbar's display, so where no toolbar is running the external pager is used regardless.os.md,prompt.mdandinitialization.md, and a changelog entry.Commits
Add a built-in pager which keeps the bottom toolbar visible. The exact code, tests and docs which were removed from Consolidated toolbar during command execution #1745, restored unchanged.
Parse paged output a line at a time, when the line is shown. New. The pager used to parse the whole text with prompt-toolkit's ANSI parser before showing any of it, twice, and that parser makes a fragment of every character:
Keep the pager's pages out of iTerm2's scrollback. New, from testing in iTerm2 (
242d9d96). With its default settings iTerm2 saves a screen which is erased from its top-left corner to the scrollback, the alternate screen included. A prompt-toolkit renderer erases its screen from there whenever it is reset: when the pager closes, each time the terminal is resized while it is open, and when Ctrl-Z lends the terminal. Each left the pager's page, its key summary and a toolbar row in the scrollback, and growing the window brought them back into view. A plain prompt-toolkit full-screen application does the same;lessdoes not. The renderer always erases from the first column of a row, so while the pager has its screen a VT100 output erases the row and then from its second column to the end of the screen, which iTerm2 does not save. The cells erased and the cursor afterwards are the same. In iTerm2 3.7.7, one pager session with three resizes and a Ctrl-Z left 226 paged rows, 11 key summaries and 11 toolbar rows in the scrollback before, and none after.What I would most like reviewed
_AnsiLexerand_styled_and_plain()inpager.py. Parsing one line by itself has to give what parsing everything gave. Two things make that true: a style still in effect at the start of a line is put in front of it, and everything else the parser removes from text (which could have swallowed a newline) is removed beforehand with a regular expression written to match the parser. The tests compare the two on a list of awkward inputs and on 3,000 random ones, and a one-off run of 200,000 random inputs found no difference. The one intended difference: readline's zero-width markers (\x01…\x02) are now left out instead of being passed through to the terminal.CommandToolbar.page(). It reassignslayout,key_bindings,editing_modeandfull_screenon a running application and callsrenderer.erase()andrenderer.request_absolute_cursor_position(). Is the restore complete on every exit path, including the display stopping while the pager is open?erasing_from_the_second_column(). It replaceserase_downon the output object for as long as the pager is open, and relies on the renderer only ever erasing from the first column of a row. Is that an acceptable way to work around a terminal, or should the pager avoidRenderer.erase()instead?Tests
tests/test_pager.py: the view, its keys, search, chopping, resizing, typeahead, Rich hyperlinks, and the comparisons described above.tests/test_command_toolbar.py: the pager inside the running toolbar, output which fits, redirected output, EOF while paging, and the UI-call helperpage()uses.tests/test_pipeline_job_control.py: a real-pty test which stops the pager with Ctrl-Z under an interactive bash and resumes it withfg.make check,make testandmake docs-testpass locally on macOS.The pager was also run in real terminals, each typed into, resized and read back, scrollback included: iTerm2 3.7.7, Terminal.app and tmux 3.8 on macOS, and tmux 3.3a and xterm 379 on Linux. One session each of paging, three resizes while open, Ctrl-Z with
fg, and quitting. Only iTerm2 kept the pager's pages in its scrollback, and no longer does after242d9d96; the other four never did, and behave the same with and without that commit.Not covered
morein a real Windows console.