Skip to content

Commit ff372de

Browse files
Match glob characters literally in completion and quote them for the shell (#1812)
* Fixed get_exes_in_path() when files or dirs contain brackets * Updated CHANGELOG. * Match glob characters literally in completion and search symlinked PATH entries Removed the guard which made path_complete() and get_exes_in_path() return nothing when the search text contained '*' or '?'. Since the search text now goes through glob.escape(), those characters are matched literally like any other, so names containing them can be completed. get_exes_in_path() and find_editor() no longer skip PATH entries which are symbolic links to directories. get_exes_in_path() resolves each entry so a directory reachable through more than one entry is only searched once. Documented that completion results are literal names which are not escaped for a shell, and added tests for shell_cmd_complete() with bracketed names and for PATH directories containing glob characters. * Quote completions containing glob characters and address review findings Tab completion now adds quotes to a match containing '*', '?', or '[', as it already did for a match containing a space. With the wildcard guard gone, completing '!rm my*' to an unquoted 'my*file.txt' would have let the shell expand it to every matching file. get_exes_in_path() no longer raises when PATH has an empty or relative entry and the current working directory has been removed. Only absolute entries are resolved with realpath(). find_editor() now uses shutil.which() instead of walking PATH by hand. As a side effect it can find 'edit' on Windows, which the old search never could because it did not try any file extensions. Documented that a leading '~' is the exception to literal matching, promoted the 'Special Characters in Paths' docs section to H2, and moved the test helper which creates fake executables into conftest.py. * Only quote path and shell command completions for the shell The glob-character quoting added in the previous commit applied to every completer. That broke commands which preserve quotes when one of their choices had a glob character, left a stray quote when completing an alias with a glob character in its name, and quoted every shell command when an executable named '[' was among the matches. path_complete() and shell_cmd_complete() now mark their results, and only marked results are quoted for a shell. The check covers the characters which are special to the platform's shell rather than only glob characters, a '[' only counts when a ']' follows it, and single quotes are used on POSIX when a name has a character which a shell still expands within double quotes. All other completions are quoted only for a space, as before. find_editor() goes back to searching the PATH directories itself. shutil.which() also searched the current directory on Windows and fell back to a default path when PATH was not set. The Windows editor list now names edit.exe so it can still be found. find_editor() and get_exes_in_path() share one helper for reading PATH. * Only quote completions for the shell where they are passed to one Quoting every path_complete() result for shell-special characters reached commands which never involve a shell. A command taking its raw argument, with path_complete as its completer, received the quotes literally for a name like notes[2026].txt and could no longer open the file. The quoting is now decided where cmd2 knows a shell is the consumer: the arguments of the shell command and everything after a pipe. Everywhere else only a match with a space is quoted, as before. Within that scope a '[' is special even with no ']' after it, since zsh rejects it as a bad pattern, a leading '=' is special on POSIX, and the Windows list follows the characters cmd.exe documents as requiring quotes. The private Completions._quote_for_shell field no longer takes part in comparisons. * Stop resolving PATH entries and fold shell marking into one helper os.path.realpath() can raise for a PATH entry which can't be reached, such as one on an unavailable network share on Windows. Since find_editor() runs when cmd2 is imported, that could keep cmd2 from being imported. The PATH entries are now used as written. Executable names were already de-duplicated, so the only cost is searching a directory twice when it is listed under two names. The places which mark completions as being passed to a shell now share one helper, Cmd._for_shell(). Corrected the docs for how the quote character is chosen when several matches are offered, and documented that a macro which runs a shell command is not quoted for the shell. * Document why an empty PATH entry is kept as the current directory --------- Co-authored-by: Todd Leonhardt <2812720+tleonhardt@users.noreply.github.com>
1 parent 05128bf commit ff372de

8 files changed

Lines changed: 659 additions & 43 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,21 @@
33
- Bug Fixes
44
- Fixed file path completion for names containing square brackets, including bracketed working
55
directories and home directories.
6+
- Fixed `get_exes_in_path()` and shell command completion when executable names or `PATH`
7+
directories contain square brackets.
8+
- File path completion, shell command completion, and `get_exes_in_path()` now match `*` and `?`
9+
literally instead of returning nothing when the text being completed contains one, so names
10+
containing those characters can be completed.
11+
- Tab completion now adds quotes when a match has a character which is special to a shell, such
12+
as `*`, `?`, `(`, `&`, or `$`, if what is being completed is passed to one. That is the case
13+
for the arguments of the `shell` command and for everything after a pipe. Everywhere else,
14+
only a match with a space is quoted, as before. See the
15+
[Special Characters in Paths](docs/features/completion.md#special-characters-in-paths)
16+
documentation for details.
17+
- `get_exes_in_path()`, `find_editor()`, and shell command completion no longer skip `PATH`
18+
entries which are symbolic links to directories.
19+
- `find_editor()` now finds the `edit` editor on Windows. It was the preferred default editor
20+
there but could never be found because it was searched for without its `.exe` extension.
621
- A command which prints through a `cmd2` console of its own, such as a `Cmd2GeneralConsole`
722
showing a Rich progress bar, no longer ends with a `BrokenPipeError` message when its output
823
is piped to something that exits early, as `head` does. `cmd2` now ends the command quietly

‎cmd2/cmd2.py‎

Lines changed: 43 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2385,6 +2385,10 @@ def path_complete(
23852385
) -> Completions:
23862386
"""Perform completion of local file system paths.
23872387
2388+
Apart from a leading ``~``, which refers to a user's home directory, every character in ``text`` is
2389+
matched literally. Glob characters like ``*``, ``?``, and ``[`` are not expanded, and they are not
2390+
escaped in the results.
2391+
23882392
:param text: the string prefix we are attempting to match (all matches must begin with it)
23892393
:param line: the current input line with leading whitespace removed
23902394
:param begidx: the beginning index of the prefix text
@@ -2399,25 +2403,19 @@ def path_complete(
23992403
if endidx == len(line) or (endidx < len(line) and line[endidx] != os.path.sep):
24002404
add_trailing_sep_if_dir = True
24012405

2402-
# Used to replace cwd in the final results
2406+
# Used to replace CWD in the final results
24032407
cwd = os.getcwd()
24042408
cwd_added = False
24052409

24062410
# Used to replace expanded user path in final result
24072411
orig_tilde_path = ""
24082412
expanded_tilde_path = ""
24092413

2410-
# If the search text is blank, then search in the CWD for *
2414+
# If the search text is blank, then complete all entries in the CWD
24112415
if not text:
24122416
search_str = os.path.join(os.getcwd(), "")
24132417
cwd_added = True
24142418
else:
2415-
# Purposely don't match any path containing wildcards
2416-
wildcards = ["*", "?"]
2417-
for wildcard in wildcards:
2418-
if wildcard in text:
2419-
return Completions()
2420-
24212419
# Start the search string
24222420
search_str = text
24232421

@@ -2436,7 +2434,7 @@ def path_complete(
24362434
orig_tilde_path = text[:sep_index]
24372435
expanded_tilde_path = os.path.expanduser(orig_tilde_path)
24382436

2439-
# If the search text does not have a directory, then use the cwd
2437+
# If the search text does not have a directory, then use the CWD
24402438
elif not os.path.dirname(text):
24412439
search_str = os.path.join(os.getcwd(), search_str)
24422440
cwd_added = True
@@ -2465,7 +2463,7 @@ def path_complete(
24652463
matches[index] += os.path.sep
24662464
display_matches[index] += os.path.sep
24672465

2468-
# Remove cwd if it was added to match the text prompt-toolkit expects
2466+
# Remove CWD if it was added to match the text prompt-toolkit expects
24692467
if cwd_added:
24702468
to_replace = cwd if cwd == os.path.sep else cwd + os.path.sep
24712469
matches = [cur_path.replace(to_replace, "", 1) for cur_path in matches]
@@ -2501,7 +2499,7 @@ def shell_cmd_complete(
25012499
if not complete_blank and not text:
25022500
return Completions()
25032501

2504-
# If there are no path characters in the search text, then do shell command completion in the user's path
2502+
# If the search text is a bare command name (no '~' or path separator), complete executables from PATH
25052503
if not text.startswith("~") and os.path.sep not in text:
25062504
items = [CompletionItem(exe) for exe in utils.get_exes_in_path(text)]
25072505
return Completions(items=items)
@@ -2536,6 +2534,9 @@ def _redirect_complete(self, text: str, line: str, begidx: int, endidx: int, com
25362534
# True when command line contains any redirection tokens
25372535
has_redirection = False
25382536

2537+
# True when the first redirection token is a pipe, which means everything after it is passed to a shell
2538+
piped_to_shell = False
2539+
25392540
# Keep track of state while examining tokens
25402541
in_pipe = False
25412542
in_file_redir = False
@@ -2546,6 +2547,8 @@ def _redirect_complete(self, text: str, line: str, begidx: int, endidx: int, com
25462547
for cur_token in raw_tokens:
25472548
# Process redirection tokens
25482549
if cur_token in constants.REDIRECTION_TOKENS:
2550+
if not has_redirection:
2551+
piped_to_shell = cur_token == constants.REDIRECTION_PIPE
25492552
has_redirection = True
25502553

25512554
# Check if we are at a pipe
@@ -2578,11 +2581,10 @@ def _redirect_complete(self, text: str, line: str, begidx: int, endidx: int, com
25782581

25792582
prior_token = cur_token
25802583

2581-
if do_shell_completion:
2582-
return self.shell_cmd_complete(text, line, begidx, endidx)
2583-
2584-
if do_path_completion:
2585-
return self.path_complete(text, line, begidx, endidx)
2584+
if do_shell_completion or do_path_completion:
2585+
completer = self.shell_cmd_complete if do_shell_completion else self.path_complete
2586+
completions = completer(text, line, begidx, endidx)
2587+
return self._for_shell(completions) if piped_to_shell else completions
25862588

25872589
# If there were redirection strings anywhere on the command line, then we
25882590
# are no longer completing for the current command
@@ -2729,7 +2731,13 @@ def _perform_completion(
27292731
if not completion_token_quote:
27302732
matches = completions.to_strings()
27312733

2732-
if any(" " in match for match in matches):
2734+
# Matches which are being passed to a shell need quotes if any of them has a character special to it
2735+
if completions._quote_for_shell:
2736+
_quote_char = utils._shell_quote_char(matches)
2737+
_add_opening_quote = bool(_quote_char)
2738+
2739+
# Everything else only needs quotes to keep a match with a space as a single token
2740+
elif any(" " in match for match in matches):
27332741
_add_opening_quote = True
27342742

27352743
# Determine best quote (single vs double) based on text content
@@ -5036,13 +5044,29 @@ def do_set(self, args: argparse.Namespace) -> None:
50365044
@classmethod
50375045
def _build_shell_parser(cls) -> Cmd2ArgumentParser:
50385046
shell_parser = argparse_utils.DEFAULT_ARGUMENT_PARSER(description="Execute a command as if at the OS prompt.")
5039-
shell_parser.add_argument("command", help="the command to run", completer=cls.shell_cmd_complete)
5047+
shell_parser.add_argument("command", help="the command to run", completer=cls._complete_shell_command)
50405048
shell_parser.add_argument(
5041-
"command_args", nargs=argparse.REMAINDER, help="arguments to pass to command", completer=cls.path_complete
5049+
"command_args", nargs=argparse.REMAINDER, help="arguments to pass to command", completer=cls._complete_shell_arg
50425050
)
50435051

50445052
return shell_parser
50455053

5054+
@staticmethod
5055+
def _for_shell(completions: Completions) -> Completions:
5056+
"""Mark completions as being for part of the command line which is passed to a shell.
5057+
5058+
This is what gets them quoted when one of them has a character which is special to that shell.
5059+
"""
5060+
return dataclasses.replace(completions, _quote_for_shell=True)
5061+
5062+
def _complete_shell_command(self, text: str, line: str, begidx: int, endidx: int) -> Completions:
5063+
"""Complete the command which do_shell passes to a shell."""
5064+
return self._for_shell(self.shell_cmd_complete(text, line, begidx, endidx))
5065+
5066+
def _complete_shell_arg(self, text: str, line: str, begidx: int, endidx: int) -> Completions:
5067+
"""Complete an argument of the command which do_shell passes to a shell."""
5068+
return self._for_shell(self.path_complete(text, line, begidx, endidx))
5069+
50465070
# Preserve quotes since we are passing these strings to the shell
50475071
@with_argparser(_build_shell_parser, preserve_quotes=True)
50485072
def do_shell(self, args: argparse.Namespace) -> None:

‎cmd2/completion.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -305,3 +305,7 @@ class Completions(CompletionResultsBase):
305305

306306
# The quote character to use if adding an opening or closing quote to the matches.
307307
_quote_char: str = ""
308+
309+
# Whether the matches are for part of the command line which is passed to a shell. If they are,
310+
# then quotes are also added when a match has a character which is special to that shell.
311+
_quote_for_shell: bool = field(default=False, compare=False)

‎cmd2/utils.py‎

Lines changed: 54 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
Generator,
1818
Iterable,
1919
MutableSequence,
20+
Sequence,
2021
)
2122
from difflib import SequenceMatcher
2223
from typing import (
@@ -323,6 +324,36 @@ def expand_user_in_tokens(tokens: MutableSequence[str]) -> None:
323324
tokens[index] = expand_user(tokens[index])
324325

325326

327+
# Matches a character which is special to a POSIX shell. A '[' counts even when no ']' follows it since
328+
# some shells, like zsh, reject that as a bad pattern. A '#' and a '=' only count at the start of a word.
329+
_POSIX_SPECIAL_RE = re.compile(r"""[ \t"';|<>&()*?\[{}$`\\]|^[#=]""")
330+
331+
# Matches a character which cmd.exe documents as requiring quotes. A '~' is left out because it is not
332+
# special to cmd.exe itself, and quoting it would quote every path below a home directory.
333+
_WINDOWS_SPECIAL_RE = re.compile(r"""[ \t&()\[\]{}^=;!'+,`]""")
334+
335+
336+
def _shell_quote_char(matches: Sequence[str]) -> str:
337+
"""Return the quote character which completions being passed to a shell need to be wrapped in.
338+
339+
:param matches: the completions being offered for part of a command line which is passed to a shell
340+
:return: the quote character to use, or an empty string if none of the matches need quotes
341+
"""
342+
if sys.platform.startswith("win"):
343+
# cmd.exe only treats double quotes as quotes, and a Windows file name can't contain one
344+
return '"' if any(_WINDOWS_SPECIAL_RE.search(match) for match in matches) else ""
345+
346+
if not any(_POSIX_SPECIAL_RE.search(match) for match in matches):
347+
return ""
348+
349+
# A quote character within a match has to be quoted with the other quote character
350+
if any("'" in match for match in matches):
351+
return '"'
352+
353+
# Only use single quotes when a shell would still expand something within double quotes
354+
return "'" if any(char in match for match in matches for char in '"$`\\') else '"'
355+
356+
326357
def _quote_shell_arg(arg: str) -> str:
327358
"""Quote a string so the OS shell treats it as one literal argument.
328359
@@ -345,6 +376,21 @@ def _quote_shell_arg(arg: str) -> str:
345376
return shlex.quote(arg)
346377

347378

379+
def _get_path_dirs() -> set[str]:
380+
"""Return the directories listed in the PATH environment variable.
381+
382+
The entries are returned as they are written. Resolving them would avoid searching a directory twice
383+
when it is listed under two names, but doing so can raise an exception for an entry which can't be reached.
384+
385+
An empty entry is kept because a POSIX shell searches the current working directory for one. A PATH which
386+
is set to an empty string is a single empty entry, so it is not the same as a PATH which is not set.
387+
388+
:return: the directories, which is an empty set if PATH is not set
389+
"""
390+
env_path = os.getenv("PATH")
391+
return set() if env_path is None else set(env_path.split(os.path.pathsep))
392+
393+
348394
def find_editor() -> str | None:
349395
"""Set cmd2.Cmd.DEFAULT_EDITOR. If EDITOR env variable is set, that will be used.
350396
@@ -354,15 +400,12 @@ def find_editor() -> str | None:
354400
editor = os.environ.get("EDITOR")
355401
if not editor:
356402
if sys.platform[:3] == "win":
357-
editors = ["edit", "code.cmd", "notepad++.exe", "notepad.exe"]
403+
editors = ["edit.exe", "code.cmd", "notepad++.exe", "notepad.exe"]
358404
else:
359405
editors = ["vim", "vi", "emacs", "nano", "pico", "joe", "code", "subl", "gedit", "kate"]
360406

361-
# Get a list of every directory in the PATH environment variable and ignore symbolic links
362-
env_path = os.getenv("PATH")
363-
paths = [] if env_path is None else [p for p in env_path.split(os.path.pathsep) if not os.path.islink(p)]
364-
365-
for possible_editor, path in itertools.product(editors, paths):
407+
# Use the most preferred editor which is in any of the PATH directories
408+
for possible_editor, path in itertools.product(editors, _get_path_dirs()):
366409
editor_path = os.path.join(path, possible_editor)
367410
if os.path.isfile(editor_path) and os.access(editor_path, os.X_OK):
368411
if sys.platform[:3] == "win":
@@ -408,26 +451,19 @@ def files_from_glob_patterns(patterns: Iterable[str], access: int = os.F_OK) ->
408451
def get_exes_in_path(starts_with: str) -> list[str]:
409452
"""Return names of executables in a user's path.
410453
454+
Every character in ``starts_with`` is matched literally. Glob characters like ``*``, ``?``, and ``[``
455+
are not expanded.
456+
411457
:param starts_with: what the exes should start with. leave blank for all exes in path.
412458
:return: a list of matching exe names
413459
"""
414-
# Purposely don't match any executable containing wildcards
415-
wildcards = ["*", "?"]
416-
for wildcard in wildcards:
417-
if wildcard in starts_with:
418-
return []
419-
420-
# Get a list of every directory in the PATH environment variable and ignore symbolic links
421-
env_path = os.getenv("PATH")
422-
paths = [] if env_path is None else [p for p in env_path.split(os.path.pathsep) if not os.path.islink(p)]
423-
424460
# Use a set to store exe names since there can be duplicates
425461
exes_set = set()
426462

427463
# Find every executable file in the user's path that matches the pattern
428-
for path in paths:
464+
for path in _get_path_dirs():
429465
full_path = os.path.join(path, starts_with)
430-
matches = files_from_glob_pattern(full_path + "*", access=os.X_OK)
466+
matches = files_from_glob_pattern(glob.escape(full_path) + "*", access=os.X_OK)
431467

432468
for match in matches:
433469
exes_set.add(os.path.basename(match))

‎docs/features/completion.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,52 @@ line of code similar to the following to your class which inherits from `cmd2.Cm
3333
complete_bar = functools.partialmethod(cmd2.Cmd.path_complete, path_filter=os.path.isdir)
3434
```
3535

36+
## Special Characters in Paths
37+
38+
Path and shell command completion match the text being completed literally. Glob characters such as
39+
`*`, `?`, and `[` are never expanded, so a file named `my*file.txt` or `notes[2026].txt` completes
40+
like any other. The one exception is a leading `~`, which refers to a user's home directory as it
41+
does in a shell.
42+
43+
Completion results are literal names too. `cmd2` does not escape them, because most commands use a
44+
path directly in Python, where the literal name is exactly what is wanted. The only thing `cmd2`
45+
adds is quotes around a match which contains a space, so it stays a single argument.
46+
47+
### Paths Passed to a Shell
48+
49+
There are two places where `cmd2` knows that what is being completed will be passed to a shell: the
50+
arguments of the [shell](./builtin_commands.md#shell) command, including its `!` shortcut, and
51+
everything after a [pipe](./redirection.md). There `cmd2` also adds quotes when a match has a
52+
character which is special to that shell, so the shell does not interpret it. For example,
53+
completing `!ls my*` gives:
54+
55+
```text
56+
(Cmd) !ls "my*file.txt"
57+
```
58+
59+
On POSIX systems the characters which get a match quoted are a space, a quote character, `*`, `?`,
60+
`[`, `{`, `}`, `(`, `)`, `<`, `>`, `|`, `&`, `;`, `$`, a backtick, a backslash, and a leading `#` or
61+
`=`. Single quotes are used when a match has a `$`, a backtick, a backslash, or a double quote in
62+
it, since a shell still interprets those within double quotes. On Windows the characters are the
63+
ones `cmd.exe` documents as requiring quotes, other than `~`, and the quotes are always double
64+
quotes.
65+
66+
There are limits to this:
67+
68+
- One pair of quotes goes around the whole argument, and it is chosen before you pick a match. So
69+
when several matches are offered and only one of them needs quotes, they all get them. An
70+
executable named `[` is common, so completing a shell command which could match it quotes every
71+
match.
72+
- For the same reason, one quote character is used for all the matches being offered. If any of them
73+
has a single quote in it, double quotes are used, and a `$`, a backtick, or a backslash in any of
74+
the matches is then still interpreted by a shell.
75+
- A `%` in a match is still interpreted by `cmd.exe` on Windows.
76+
- A [macro](./shortcuts_aliases_macros.md#macros) which runs a shell command is not covered. Its
77+
arguments are completed as paths, but they are only quoted for a space. An alias for the `shell`
78+
command is covered.
79+
- Your own commands are not covered, even if they pass their arguments to a shell. Quote or escape
80+
those arguments in the command.
81+
3682
## Included Tab Completion Functions
3783

3884
[cmd2.Cmd][] provides the following tab completion functions

‎tests/conftest.py‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,12 @@
33
import os
44
import subprocess
55
import sys
6-
from collections.abc import Callable
6+
from collections.abc import (
7+
Callable,
8+
Iterable,
9+
)
710
from contextlib import redirect_stderr
11+
from pathlib import Path
812
from typing import (
913
TYPE_CHECKING,
1014
ParamSpec,
@@ -61,6 +65,14 @@ def normalize(block: str) -> list[str]:
6165
return [line.rstrip() for line in block.splitlines()]
6266

6367

68+
def make_exes(directory: Path, names: Iterable[str]) -> None:
69+
"""Create an executable file in directory for each name."""
70+
for name in names:
71+
exe = directory / name
72+
exe.touch()
73+
exe.chmod(0o755)
74+
75+
6476
def run_cmd(app: cmd2.Cmd, cmd: str) -> tuple[list[str], list[str]]:
6577
"""Run the command in app and return what it writes to app.stdout and sys.stderr."""
6678

0 commit comments

Comments
 (0)