From 8bfcc6ea7102f83f2340b4d200c002105523588a Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Thu, 29 Oct 2026 05:07:21 +1100 Subject: [PATCH 1/9] Turn on the flag u by default, and add -U to turn it off --- README.md | 9 +++++---- src/cli.ts | 16 +++++++++++++--- src/engine.ts | 17 ++++++++++++++++- test/cli/debug.test.js | 2 +- test/cli/run.sh | 2 +- test/engine/crossover.test.js | 4 ++-- test/engine/unicode.test.js | 19 ++++++++++++++----- test/engine/utf8.test.js | 8 ++++---- test/engine/void-section.test.js | 6 +++--- test/fix/683.test.js | 7 ++++--- test/fix/742.test.js | 2 +- test/fix/854.test.js | 14 +++++++++----- 12 files changed, 73 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 507150ae..bc4a33ea 100644 --- a/README.md +++ b/README.md @@ -322,8 +322,9 @@ If planned renames clash, the process is halted and no change is applied to any | `-G` | **`--void-global`** Void global search (stop looking after the first match). [boolean] | | `-s` | **`--dot-all`** Have `.` also match newline. [boolean] | | `-M` | **`--void-multiline`** Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] | -| `-u` | **`--unicode`** Treat pattern as a sequence of unicode code points. [boolean] | -| | **`--flags`** The flags of the regex, like `gimsv`, in place of the defaults: `gim`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s or -u. [string] | +| `-u` | **`--unicode`** Treat pattern as a sequence of unicode code points. It is the default, so it is only needed with `-E regex+`, where it gives `u` in place of `v`. [boolean] | +| `-U` | **`--void-unicode`** Void unicode: read the pattern as UTF-16 parts, without the strict syntax of the flag `u`, as before v8. [boolean] | +| | **`--flags`** The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U. [string] | | `-e` | **`--encoding`** Encoding of files/piped data. [default: "utf8"] | | `-E` | **`--engine`** The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun. [choices: "rr", "default", "regex+", "runtime"] [default: "rr"] | | `-q` | **`--quiet`** Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown. [boolean] | @@ -390,7 +391,7 @@ Files identified for potential processing are identified by: - Patterns are described as javascript notation regex - Pattern defaults to global multiline case-insensitive search - Supports regex lookaheads in the pattern -- Supports the extended syntax of [Regex+](https://github.com/slevithan/regex) in the pattern: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g` with `(?(DEFINE)…)`. A pattern that works in the native engine of the runtime works the same, with one difference: `\g` calls the group, where the native engine reads it as the text `g` (use `-E runtime` for that). Regex+ emulates atomic groups and possessive quantifiers with a lookahead and a backreference, and a lookbehind reads from right to left, so inside a lookbehind they look forward in place of back, and match in the wrong place with no error: `(?<=(?>a))` finds the places in front of an `a`, not after it. A subroutine needs a named group that exists, as Regex+ has no numbered ones like `\g<1>`. A subroutine copies the groups inside the group it calls, and an unnamed one there gets a number of its own: in `(?

(a)(b))\g

`, `$4` and `$5` are the copies of `$2` and `$3`, and `--output-match` prints them too. Name the groups inside, or leave them out, to avoid it. Without `-u`, the native engine reads `[[]` as a class with a `[` in it, but Regex+ sees the class as still open, so the extended syntax after it is not read: `[[](?>a)` stops the run, and `[\[](?>a)` works. Counting the groups Regex+ adds, a replacement text can only reach the first 99 groups, as in javascript. With `-E regex+`, `-u` gives `u` in place of `v`. The flags `u` and `v` can not go together. +- Supports the extended syntax of [Regex+](https://github.com/slevithan/regex) in the pattern: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g` with `(?(DEFINE)…)`. A pattern that works in the native engine of the runtime, with the same flags, works the same, with one difference: `\g` calls the group, where the native engine reads it as the text `g` (use `-E runtime` for that). Regex+ emulates atomic groups and possessive quantifiers with a lookahead and a backreference, and a lookbehind reads from right to left, so inside a lookbehind they look forward in place of back, and match in the wrong place with no error: `(?<=(?>a))` finds the places in front of an `a`, not after it. A subroutine needs a named group that exists, as Regex+ has no numbered ones like `\g<1>`. A subroutine copies the groups inside the group it calls, and an unnamed one there gets a number of its own: in `(?

(a)(b))\g

`, `$4` and `$5` are the copies of `$2` and `$3`, and `--output-match` prints them too. Name the groups inside, or leave them out, to avoid it. With `-U`, the native engine reads `[[]` as a class with a `[` in it, but Regex+ sees the class as still open, so the extended syntax after it is not read: `[[](?>a)` stops the run, and `[\[](?>a)` works. Counting the groups Regex+ adds, a replacement text can only reach the first 99 groups, as in javascript. With `-E regex+`, `-u` gives `u` in place of `v`. The flags `u` and `v` can not go together. - Supports backreference to matching groups in the replacement - Data can be piped in, both as the text to search and as the replacement - The names of the changed files are printed to stdout, so they can be piped on (to stderr with `--output` and `--output-match`) @@ -464,7 +465,7 @@ Each part of a run is a stack of layers. Each layer is optional, and works on wh 1. The pattern as typed. `€` is `$` and `§` is `\`. 2. `--literal` (`-L`) makes it plain text, so `.` is a dot. -3. `--void-ignore-case` (`-I`), `--void-multiline` (`-M`), `--dot-all` (`-s`), `--unicode` (`-u`) and `--void-global` (`-G`) decide how the regex works. +3. `--void-ignore-case` (`-I`), `--void-multiline` (`-M`), `--dot-all` (`-s`), `--void-unicode` (`-U`) and `--void-global` (`-G`) decide how the regex works, or `--flags` sets all the flags. The flag `u` is on by default, so an emoji is one char, and the syntax is strict: an escape with no meaning like `\"` or `\-`, or a `{`, `}` or `]` on its own, stops the run, and the error says when `-U` would make it work. ```bash > echo -n 'a.b' | rexreplace '.' '_' diff --git a/src/cli.ts b/src/cli.ts index c2142faf..1e1e7c2e 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -118,16 +118,26 @@ const yargs = createRequire(import.meta.url)('yargs') .alias('M', 'void-multiline') .boolean('u') - .describe('u', 'Treat pattern as a sequence of unicode code points.') + .describe( + 'u', + 'Treat pattern as a sequence of unicode code points. It is the default, so it is only needed with `-E regex+`, where it gives `u` in place of `v`.' + ) .alias('u', 'unicode') + .boolean('U') + .describe( + 'U', + 'Void unicode: read the pattern as UTF-16 parts, without the strict syntax of the flag `u`, as before v8.' + ) + .alias('U', 'void-unicode') + .string('flags') - .conflicts('flags', ['I', 'G', 'M', 's', 'u']) + .conflicts('flags', ['I', 'G', 'M', 's', 'u', 'U']) // Like the other options, the last one wins when it is given twice .coerce('flags', value => [].concat(value).pop()) .describe( 'flags', - 'The flags of the regex, like `gimsv`, in place of the defaults: `gim`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s or -u.' + 'The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U.' ) .default('e', 'utf8') diff --git a/src/engine.ts b/src/engine.ts index d90a8bcc..a6cd05ab 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -748,7 +748,7 @@ function regexFlags(config, regexPlus) { (config.voidIgnoreCase ? '' : 'i') + (config.voidMultiline ? '' : 'm') + (config.dotAll ? 's' : '') + - (regexPlus ? (config.unicode ? 'uxn' : 'vxn') : config.unicode ? 'u' : '') + (regexPlus ? (config.unicode ? 'uxn' : 'vxn') : config.voidUnicode ? '' : 'u') ); } @@ -805,10 +805,25 @@ function compileRegex(pattern, config) { } return {regex: new RegExp(pattern, flags), hiddenCaptures: [], namedOnly: false}; } catch (e) { + // The flag u is on by default, and it is strict about the syntax, so a pattern from before v8 can fail on it + if (!regexPlus && null == config.flags && worksWithoutUnicode(pattern, flags, engine)) { + return panic(e.message, 'It works without the flag u, which -U (--void-unicode) turns off.'); + } return panic(e.message); } } +function worksWithoutUnicode(pattern, flags, engine) { + const legacy = flags.replace('u', ''); + try { + return Boolean( + new RegExp('rr' === engine ? extendedSyntax(pattern, legacy).pattern : pattern, legacy) + ); + } catch (e) { + return false; + } +} + // Only the transforms of Regex+ run, in its own order, and not its flags, so a pattern that works natively gives the same regex. // A transform only runs when its syntax is outside a character class, as Regex+ does not always look, like for [(?>]\1 (#858). function extendedSyntax(pattern, flags) { diff --git a/test/cli/debug.test.js b/test/cli/debug.test.js index 534b7535..0891bfa2 100644 --- a/test/cli/debug.test.js +++ b/test/cli/debug.test.js @@ -19,7 +19,7 @@ describe('--debug', () => { engine: 'rr', hiddenCaptures: [], search: 'o', - flags: 'gim', + flags: 'gimu', replacement: 'x', replacementSrc: null, }); diff --git a/test/cli/run.sh b/test/cli/run.sh index 85b3ce6c..ecebb957 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -138,7 +138,7 @@ assert "rexreplace '.€' '€' my.file -o --void-euro" 'foobar' reset echo foo[bar] > my.file assert "rexreplace '[§]]' '[' my.file -o" 'foo[bar[' -assert "rexreplace '[§]]' '[' my.file -o --void-section" 'foo[bar]' +assert "rexreplace '[§]]' '[' my.file -o --void-section -U" 'foo[bar]' reset rexreplace 'b' '§n' my.file assert "cat my.file" "foo\nar" diff --git a/test/engine/crossover.test.js b/test/engine/crossover.test.js index 7f5bdeb2..dd799d0d 100644 --- a/test/engine/crossover.test.js +++ b/test/engine/crossover.test.js @@ -4,7 +4,7 @@ import fs from 'fs'; import {rr} from '../helper.js'; describe('regex flags', () => { - const flags = ['voidIgnoreCase', 'voidGlobal', 'voidMultiline', 'dotAll', 'unicode']; + const flags = ['voidIgnoreCase', 'voidGlobal', 'voidMultiline', 'dotAll', 'voidUnicode']; const text = 'Foo\nfoo 🦖\r\nFOO bar\n'; const patterns = ['foo', '^.', '.$', 'o.f', '^f.*o$', '\\S+$', 'O\\s', '.']; @@ -22,7 +22,7 @@ describe('regex flags', () => { (config.voidIgnoreCase ? '' : 'i') + (config.voidMultiline ? '' : 'm') + (config.dotAll ? 's' : '') + - (config.unicode ? 'u' : ''); + (config.voidUnicode ? '' : 'u'); for (const pattern of patterns) { expect((await rr(pattern, '<$&>', {pipedData: text, ...config})).out, pattern).toBe( diff --git a/test/engine/unicode.test.js b/test/engine/unicode.test.js index 9beea484..088b62df 100644 --- a/test/engine/unicode.test.js +++ b/test/engine/unicode.test.js @@ -7,13 +7,14 @@ describe('--unicode', () => { expect((await rr('^.$', 'x', {pipedData: '🦖', unicode: true})).out).toBe('x'); }); - test('is not the default', async () => { - expect((await rr('^.$', 'x', {pipedData: '🦖'})).out).toBe('🦖'); + test('is the default, and --void-unicode turns it off', async () => { + expect((await rr('^.$', 'x', {pipedData: '🦖'})).out).toBe('x'); + expect((await rr('^.$', 'x', {pipedData: '🦖', voidUnicode: true})).out).toBe('🦖'); }); test('understands code point escapes', async () => { expect((await rr('\\u{1F996}', 'x', {pipedData: 'a🦖b', unicode: true})).out).toBe('axb'); - expect((await rr('\\u{1F996}', 'x', {pipedData: 'a🦖b'})).out).toBe('a🦖b'); + expect((await rr('\\u{1F996}', 'x', {pipedData: 'a🦖b', voidUnicode: true})).out).toBe('a🦖b'); }); test('understands unicode property escapes', async () => { @@ -36,13 +37,21 @@ describe('--unicode', () => { }); test('is strict about escapes that have no meaning', async () => { - expect((await rr('\\-', 'x', {pipedData: 'a-b'})).out).toBe('axb'); - const result = await rr('\\-', 'x', {pipedData: 'a-b', unicode: true}); + expect((await rr('\\-', 'x', {pipedData: 'a-b', voidUnicode: true})).out).toBe('axb'); + const result = await rr('\\-', 'x', {pipedData: 'a-b'}); expect(result.exit).toBe(3); expect(result.err).toContain('Invalid regular expression'); expect(result.out).toBe(''); }); + test('tells that a pattern from before works without it', async () => { + const result = await rr('\\-', 'x', {pipedData: 'a-b'}); + expect(result.err).toContain( + '\n It works without the flag u, which -U (--void-unicode) turns off.\n' + ); + expect((await rr('(', 'x', {pipedData: 'a'})).err).not.toContain('-U'); + }); + describe('combined', () => { test('with files', async () => { fs.writeFileSync('my.file', 'a🦖b'); diff --git a/test/engine/utf8.test.js b/test/engine/utf8.test.js index ac2c2e64..a3946bb1 100644 --- a/test/engine/utf8.test.js +++ b/test/engine/utf8.test.js @@ -15,13 +15,13 @@ describe('#735 UTF-8 in the pattern', () => { expect(asUtf8((await rr('🦖', 'x', {pipedData: text})).out)).toBe('æ 中 x 👩‍💻\r\n'); }); - test('. takes half an emoji without --unicode, which then shows as �', async () => { - const out = (await rr('.', 'x', {pipedData: '🦖a', voidGlobal: true})).out; + test('. takes half an emoji with --void-unicode, which then shows as �', async () => { + const out = (await rr('.', 'x', {pipedData: '🦖a', voidGlobal: true, voidUnicode: true})).out; expect(asUtf8(out)).toBe('x�a'); }); - test('. takes the whole emoji with --unicode', async () => { - const config = {pipedData: '🦖a', voidGlobal: true, unicode: true}; + test('. takes the whole emoji, as --unicode is the default', async () => { + const config = {pipedData: '🦖a', voidGlobal: true}; expect(asUtf8((await rr('.', 'x', config)).out)).toBe('xa'); }); diff --git a/test/engine/void-section.test.js b/test/engine/void-section.test.js index 88e6e918..941c2fb6 100644 --- a/test/engine/void-section.test.js +++ b/test/engine/void-section.test.js @@ -19,9 +19,9 @@ describe('--void-section', () => { test('makes § a plain char in the pattern', async () => { expect((await rr('§d', '#', {pipedData: 'a1 §d', voidSection: true})).out).toBe('a1 #'); - expect((await rr('[§]]', '[', {pipedData: 'foo[bar]', voidSection: true})).out).toBe( - 'foo[bar]' - ); + expect( + (await rr('[§]]', '[', {pipedData: 'foo[bar]', voidSection: true, voidUnicode: true})).out + ).toBe('foo[bar]'); }); test('makes § a plain char in the replacement', async () => { diff --git a/test/fix/683.test.js b/test/fix/683.test.js index 25310011..c65d9963 100644 --- a/test/fix/683.test.js +++ b/test/fix/683.test.js @@ -69,9 +69,10 @@ describe('#683 --exclude-re follows the regex flags of the main search', () => { test('--unicode lets \\u{...} match in the path', async () => { fs.writeFileSync('a.file', 'foo'); fs.writeFileSync('b.file', 'foo'); - expect((await rr('foo', 'x', {includeGlob: ['*.file'], excludeRe: ['\\u{61}']})).out).toBe( - 'a.file\nb.file\n' - ); + expect( + (await rr('foo', 'x', {includeGlob: ['*.file'], excludeRe: ['\\u{61}'], voidUnicode: true})) + .out + ).toBe('a.file\nb.file\n'); fs.writeFileSync('a.file', 'foo'); fs.writeFileSync('b.file', 'foo'); expect( diff --git a/test/fix/742.test.js b/test/fix/742.test.js index 5693cbcb..fb11206c 100644 --- a/test/fix/742.test.js +++ b/test/fix/742.test.js @@ -144,7 +144,7 @@ describe('#742 the engine part of the status', () => { engine: 'rr', hiddenCaptures: [], search: 'o+', - flags: 'gi', + flags: 'giu', replacement, replacementSrc: src, }); diff --git a/test/fix/854.test.js b/test/fix/854.test.js index 3b4d2064..f3d0700b 100644 --- a/test/fix/854.test.js +++ b/test/fix/854.test.js @@ -49,13 +49,17 @@ describe('#854 the rr engine', () => { ['(?x)\\1'], ['$'], ['[[]'], - ])('gives the same regex as the native engine for %s', async pattern => { + ])('gives the same regex as the native engine for %s, without the flag u', async pattern => { const native = new RegExp(pattern, 'gim'); - expect(await compiled(pattern)).toMatchObject({search: native.source, flags: 'gim'}); + expect(await compiled(pattern, {voidUnicode: true})).toMatchObject({ + search: native.source, + flags: 'gim', + }); }); - test('only uses the u flag with -u', async () => { - expect((await compiled('a', {unicode: true})).flags).toBe('gimu'); + test('has the flag u by default, and not with -U', async () => { + expect((await compiled('a')).flags).toBe('gimu'); + expect((await compiled('a', {voidUnicode: true})).flags).toBe('gim'); }); test.each([ @@ -112,7 +116,7 @@ describe('#854 the groups Regex+ adds', () => { describe('#854 the other engines', () => { // The tests are run by Bun, so the native engine is JSC test.each([['runtime']])('%s is the native engine, without the extended syntax', async engine => { - expect(await out('(?a)\\g', 'X', 'ag', {engine})).toBe('X'); + expect(await out('(?a)\\g', 'X', 'ag', {engine, voidUnicode: true})).toBe('X'); }); test.each([['regex+']])( From e5c46d606f362434f8d709b06fd543b8664eb871 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Thu, 29 Oct 2026 05:29:03 +1100 Subject: [PATCH 2/9] Say what -u and -U do for each engine, without version numbers --- README.md | 4 ++-- src/cli.ts | 4 ++-- src/engine.ts | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index bc4a33ea..8007ad0a 100644 --- a/README.md +++ b/README.md @@ -322,8 +322,8 @@ If planned renames clash, the process is halted and no change is applied to any | `-G` | **`--void-global`** Void global search (stop looking after the first match). [boolean] | | `-s` | **`--dot-all`** Have `.` also match newline. [boolean] | | `-M` | **`--void-multiline`** Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] | -| `-u` | **`--unicode`** Treat pattern as a sequence of unicode code points. It is the default, so it is only needed with `-E regex+`, where it gives `u` in place of `v`. [boolean] | -| `-U` | **`--void-unicode`** Void unicode: read the pattern as UTF-16 parts, without the strict syntax of the flag `u`, as before v8. [boolean] | +| `-u` | **`--unicode`** Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode. [boolean] | +| `-U` | **`--void-unicode`** Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode. [boolean] | | | **`--flags`** The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U. [string] | | `-e` | **`--encoding`** Encoding of files/piped data. [default: "utf8"] | | `-E` | **`--engine`** The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun. [choices: "rr", "default", "regex+", "runtime"] [default: "rr"] | diff --git a/src/cli.ts b/src/cli.ts index 1e1e7c2e..625d4571 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -120,14 +120,14 @@ const yargs = createRequire(import.meta.url)('yargs') .boolean('u') .describe( 'u', - 'Treat pattern as a sequence of unicode code points. It is the default, so it is only needed with `-E regex+`, where it gives `u` in place of `v`.' + 'Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode.' ) .alias('u', 'unicode') .boolean('U') .describe( 'U', - 'Void unicode: read the pattern as UTF-16 parts, without the strict syntax of the flag `u`, as before v8.' + 'Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode.' ) .alias('U', 'void-unicode') diff --git a/src/engine.ts b/src/engine.ts index a6cd05ab..555c2966 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -805,7 +805,7 @@ function compileRegex(pattern, config) { } return {regex: new RegExp(pattern, flags), hiddenCaptures: [], namedOnly: false}; } catch (e) { - // The flag u is on by default, and it is strict about the syntax, so a pattern from before v8 can fail on it + // The flag u is on by default, and it is strict about the syntax, so a pattern that worked without it can fail on it if (!regexPlus && null == config.flags && worksWithoutUnicode(pattern, flags, engine)) { return panic(e.message, 'It works without the flag u, which -U (--void-unicode) turns off.'); } From e67774ce365ba48a5df48877a91b4ed89ed3af1d Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Sat, 31 Oct 2026 13:24:06 +1100 Subject: [PATCH 3/9] Try argmate from ../argmate in place of yargs --- src/cli.ts | 872 +++++++++++++++---------------------- test/cli/glob-pipe.test.js | 2 +- test/cli/pipe.test.js | 2 +- test/fix/783.test.js | 2 +- test/fix/854.test.js | 2 +- 5 files changed, 349 insertions(+), 531 deletions(-) diff --git a/src/cli.ts b/src/cli.ts index 625d4571..ffc06282 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -4,7 +4,7 @@ import * as rexreplace from './engine.ts'; import {writeTo} from './output.ts'; import {statusConfig, panic} from './status.ts'; import {alias} from './alias.ts'; -import {createRequire} from 'module'; +import argMate, {argInfo} from '../../argmate/src/argMate.ts'; // Make it visible when Bun is the runtime (like with rexreplace-bun) as the regex engine is then not the one from Node. // Added as semver build metadata (`+bun`) as a `-bun` would make it a prerelease. @@ -27,525 +27,324 @@ if (process.argv.length < 4) { [pattern, replacement] = process.argv.splice(2, 2); } -// As an ES module yargs breaks the lines of the help in the middle of words, so it is loaded as CommonJS. -// It reads process.argv when it is loaded, which is after the pattern and the replacement are taken out above. -const yargs = createRequire(import.meta.url)('yargs') - // A file can be named with only digits, like `2024`, so each glob is kept as the string that was given - .parserConfiguration({'parse-positional-numbers': false}) - .strict() - // Wrong flags are a panic, so the exit code is 3. The help and the message go to stderr, as yargs does it. - .fail((msg, err, yargs) => { - yargs.showHelp(); - console.error(msg || err.message); - process.exit(3); - }) - - .usage( +// The pattern and the replacement are taken out of argv above, so argMate only gets the rest +const argConfig = { + intro: [ 'RexReplace ' + version + - ': Regexp search and replace for files using lookahead and backreference to matching groups in the replacement. Defaults to global multiline case-insensitive search.\n\n' + - '> rexreplace pattern replacement [fileGlob|option]+' - ) - - .example(`> rexreplace 'Foo' 'xxx' myfile.md`, `'foobar' in myfile.md will become 'xxxbar'`) - .example('') - .example(`> rr xxx Foo myfile.md`, `The alias 'rr' can be used instead of 'rexreplace'`) - .example('') - - .example( - `> rexreplace '(f?(o))o(.*)' '$3$1€2' myfile.md`, - `'foobar' in myfile.md will become 'barfoo'` - ) - .example('') - .example( - `> rexreplace '^#' '##' *.md`, - `All markdown files in this dir got all headlines moved one level deeper` - ) - .example('') - .example( - `> rexreplace 'a' 'b' 'myfile.md' 'src/**/*.*' `, - `Provide multiple files or glob if needed` - ) - .version('v', 'Print rexreplace version (can be given as only argument)', version) - .alias('v', 'version') - - .boolean('V') - .describe('V', 'More chatty output') - .alias('V', 'verbose') - //.conflicts('V', 'q') - //.conflicts('V', 'Q') - - .boolean('L') - .describe('L', 'Literal string search (no regex used when searching)') - .alias('L', 'literal') - - .boolean('l') - .describe( - 'l', - 'No "smart" values in the replacement, so you can have text like `$1` and `$&` without escaping them.' - ) - .alias('l', 'literal-replacement') - - .boolean('replacement-file') - .describe( - 'replacement-file', - 'The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path.' - ) - - .boolean('trim-file') - .describe('trim-file', 'Trim the content of the replacement file.') - - .boolean('trim') - .describe('trim', 'Shorthand for --trim-pipe and --trim-file.') - - .boolean('I') - .describe('I', 'Void case insensitive search. Applies to the pattern and to all globs.') - .alias('I', 'void-ignore-case') - - .boolean('G') - .describe('G', 'Void global search (stop looking after the first match).') - .alias('G', 'void-global') - - .boolean('s') - .describe('s', 'Have `.` also match newline.') - .alias('s', 'dot-all') - - .boolean('M') - .describe( - 'M', - 'Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line.' - ) - .alias('M', 'void-multiline') - - .boolean('u') - .describe( - 'u', - 'Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode.' - ) - .alias('u', 'unicode') - - .boolean('U') - .describe( - 'U', - 'Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode.' - ) - .alias('U', 'void-unicode') - - .string('flags') - .conflicts('flags', ['I', 'G', 'M', 's', 'u', 'U']) - // Like the other options, the last one wins when it is given twice - .coerce('flags', value => [].concat(value).pop()) - .describe( - 'flags', - 'The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U.' - ) - - .default('e', 'utf8') - .alias('e', 'encoding') - .describe('e', 'Encoding of files/piped data.') - // Not .choices() as node takes the name of an encoding in any case, like `UTF8` - .check(argv => Buffer.isEncoding(argv.encoding) || `Unknown encoding: ${argv.encoding}`) - - .string('x') - .describe( - 'x', - 'Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times.' - ) - .alias('x', 'exclude-re') - - .string('X') - .describe( - 'X', - 'Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate.' - ) - .alias('X', 'exclude-glob') - - .alias('E', 'engine') - .describe( - 'E', - 'The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun.' - ) - .choices('E', ['rr', 'default', 'regex+', 'runtime']) - .default('E', 'rr') - - .boolean('q') - .describe( - 'q', - 'Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown.' - ) - .alias('q', 'quiet') - - .boolean('Q') - .describe('Q', 'Never display errors or info') - .alias('Q', 'quiet-total') - - .boolean('f') - .describe('f', 'Print the full path of the named files instead of the relative path.') - .alias('f', 'full-path') - - .boolean('name-all') - .describe( - 'name-all', - 'Print the names of all files found by the globs, also the ones where nothing was changed.' - ) - - .string('name-delimiter') - .describe( - 'name-delimiter', - 'String printed between the names of the files. Default is a line break, unless the name already ends with one.' - ) - .string('content-delimiter') - .describe( - 'content-delimiter', - 'String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one.' - ) - - .string('delimiter') - .describe( - 'delimiter', - 'Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\\t`, `\\n` and the `€` and `§` aliases.' - ) - - .boolean('null-out') - .alias('null-out', 'null') - .describe( - 'null-out', - "Shorthand for --delimiter '\\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out." - ) - - .boolean('name-delimiter-tail') - .describe( - 'name-delimiter-tail', - 'Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name.' - ) - - .boolean('content-delimiter-tail') - .describe( - 'content-delimiter-tail', - 'Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe.' - ) - - .boolean('tail') - .describe( - 'tail', - 'Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail.' - ) - - .boolean('nl') - .describe( - 'nl', - 'Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed.' - ) - - .boolean('H') - .describe('H', 'Halt on first error') - .alias('H', 'halt') - .alias('H', 'bail') - .default('H', false) - - .boolean('€') - .describe( - '€', - "Void having '€' as alias for '$'. The alias is used in all inputs, but never in globs, file names or piped data." - ) - .alias('€', 'void-euro') - - .boolean('§') - .describe( - '§', - "Void having '§' as alias for '\\'. The alias is used in all inputs, but never in globs, file names or piped data." - ) - .alias('§', 'void-section') - - .boolean('o') - .describe( - 'o', - 'Output the final result instead of saving to file. Will also output content even if no replacement has taken place.' - ) - .alias('o', 'output') - //.conflicts('o','O') - - .boolean('A') - .alias('A', 'void-async') - .describe( - 'A', - `Handle files in a synchronous flow. Good to limit memory usage when handling large files. ` + - '' - ) - - .boolean('B') - .describe( - 'B', - 'Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted.' - ) - .alias('B', 'void-backup') - - .boolean('b') - .describe('b', 'Keep a backup file of the original content. With --rename the old file is kept.') - .alias('b', 'keep-backup') - - .boolean('S') - .describe( - 'S', - 'Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate.' - ) - .alias('S', 'simulate') - - .string('sort') - .describe( - 'sort', - 'Process files sync in a sorted order chosen from 6 values: `natural` (numbers compared as numbers and case insensitive), `ascii` (by char code, so `B` comes before `a`), `modified`, `created`, `size`, or `none` (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order.' - ) - // Any case, and true and yes are natural - .coerce('sort', value => { - const name = String(value).toLowerCase(); - return 'true' === name || 'yes' === name ? 'natural' : name; - }) - .choices('sort', ['natural', 'ascii', 'modified', 'created', 'size', 'none']) - - .boolean('sort-reverse') - .describe('sort-reverse', 'Turn the order of the files around.') - - // Checks for a CI step. When one is not met, the exit code is 1. - .boolean('F') - .describe('F', 'Check that at least one file is found. The exit code is 1 if not.') - .alias('F', 'expect-files') - - .boolean('z') - .describe('z', 'Check that at least one file is changed. The exit code is 1 if not.') - .alias('z', 'expect-diff') - - .boolean('Z') - .describe('Z', 'Check that no file is changed. The exit code is 1 if one is.') - .alias('Z', 'expect-zero-diff') - - .boolean('expect-diff-all') - .describe('expect-diff-all', 'Check that all files found are changed. The exit code is 1 if not.') - .conflicts('Z', ['z', 'expect-diff-all']) - - .boolean('r') - .describe( - 'r', - 'Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots.' - ) - .alias('r', 'rename') - - .boolean('rename-path') - .describe( - 'rename-path', - 'Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made.' - ) - .conflicts('r', 'rename-path') - - .boolean('rename-force') - .describe( - 'rename-force', - 'With --rename or --rename-path, write over a file that is there already.' - ) - - .boolean('m') - .describe( - 'm', - `Output each match on a new line. ` + - `Will not replace any content but you still need to provide a dummy value (like \`_\`) as replacement parameter. The replacement is not used, so --js code is not run. ` + - `If search pattern does not contain matching groups the full match will be outputted. ` + - `If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). ` + - `` - ) - .alias('m', 'output-match') - - .boolean('T') - .alias('T', 'trim-pipe') - .describe( - 'T', - `Trim piped data before processing. ` + - `If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. ` + - '' - ) - - .boolean('R') - .alias('R', 'replacement-pipe') - .describe( - 'R', - `Replacement will be piped in. You still need to provide a dummy value (like \`_\`) as replacement parameter.` + - '' - ) - - .boolean('js-pipe') - .describe( - 'js-pipe', - `Treat the replacement as javascript, and give the code the piped data as \`pipe\`.` - ) - - .boolean('g') - .alias('g', 'glob-pipe') - .describe( - 'g', - 'Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments.' - ) - // The piped data can only be one thing - .conflicts('g', ['R', 'js-pipe']) - - .boolean('null-in') - .describe( - 'null-in', - 'The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out.' - ) - .implies('null-in', 'glob-pipe') - - .boolean('j') - .alias('j', 'js') - .alias('j', 'replacement-js') - .describe( - 'j', - `Treat replacement as javascript source code. -The statement from the last expression will become the replacement string. -Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. -The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. -At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. - -The code has access to the following variables: -\`r\` as an alias for \`require\` with both expanded to understand a relative path even if it is not starting with \`./\`, -\`fs\` from node, -\`path\` from node, -\`globs\` from npm, -\`pipe\`: the data piped into the command with --js-pipe (null if no piped data, and \`pipe_\` is \`❌ \`), -\`find\`: pattern searched for (the needle), -\`text\`: full text being searched i.e. file content or piped data (the haystack), -\`bytes\`: total size of the haystack in bytes, -\`size\`: human-friendly representation of the total size of the haystack, -\`time\`: String representing the local time when the command was invoked, -\`time_obj\`: date object representing \`time\`, -\`now\`: alias for \`time\`, -\`cwd\`: current process working dir, -\`nl\`: a new-line char, -\`_\`: a single space char (for easy string concatenation). - -The following values defaults to \`❌\` if haystack does not originate from a file: -\`file\`: contains the full path of the active file being searched (including full filename), -\`file_rel\`: contains \`file\` relative to current process working dir, -\`file_path\` and \`file_path_rel\`: the same as \`file\` and \`file_rel\`, -\`dirpath\`: contains the full path without filename of the active file being searched, -\`dirpath_rel\`: contains \`dirpath\` relative to current process working dir, -\`filename\`: is the full filename of the active file being searched without path, -\`name\`: filename of the active file being searched with no extension, -\`ext\`: extension of the filename including leading dot, -\`mtime\`: ISO inspired representation of the last local modification time of the current file, -\`ctime\`: ISO representation of the local creation time of the current file. -\`mtime_obj\`: date object representing \`mtime\`, -\`ctime_obj\`: date object representing \`ctime\`. - -The numbers: \`$n\`: the number of the file, \`$t\`: the number of files, \`$i\`: the number of the file among the files with a match so far, \`$c\`: among the files changed so far, \`$m\`: the number of the match in the file, \`$M\`: across all files. Each has cousins with zeros in front (\`$n0\` has 2 digits, \`$n00\` 3, and so on to 10) and with a space at the end (\`$n_\`). For speed, the code runs once per file, at its first match, unless it names the match, a group, \`$m\`, \`$M\` or \`rr\`. A file without a match does not run the code. - -Access a named group as \`$\` and the name: \`(?...)\` gives you \`$year\`. A group named like a number, like \`n\`, wins over the number. With the flag \`n\` of \`-E regex+\` only named groups capture, so the code has no \`$1\`, while a replacement text still reads \`$1\` as the first named group, as in javascript. - -\`rr\` has the values with long names: \`filePath\`, \`filePathRel\`, \`dirPath\`, \`dirPathRel\`, \`dirName\`, \`fileName\`, \`name\`, \`ext\`, \`bytes\`, \`size\`, \`mtime\`, \`ctime\`, \`mtimeObj\`, \`ctimeObj\`, \`fileNumber\`, \`filesTotal\`, \`matchedFileNumber\`, \`changedFileNumber\`, \`matchThisFile\`, \`matchUntilNow\`, \`text\`, \`find\`, \`stdin\`, \`cwd\`, \`time\` and \`timeObj\`, and the groups, like \`rr[1]\` and \`rr.year\`, which overwrite a name they share. Code that names \`rr\`, also in a string or a comment, runs once per match. \`print()\` and \`printErr()\`, also on \`rr\`, print to stdout and stderr, also with --quiet-total. \`pad(value, width = 3, char = '0')\` pads a value, like \`pad($1)\` for \`007\`. - -All variables, except from module, date objects, functions, \`nl\` and \`_\`, has a corresponding variable name followed by \`_\` where the content has an extra space at the end (for easy concatenation). -` - ) - - /* - .boolean('N') - .alias('N', 'void-newline') - .describe('N', - `Avoid having newline when outputting data (or when piping). `+ - `Normally . `+ - '' - ) -*/ - - /* .boolean('P') - .describe('P', "Pattern is a filename from where the pattern will be generated. If more than one line is found in the file the pattern will be defined by each line trimmed and having newlines removed followed by other all rules (like -€).)") - .alias('P', 'pattern-file') - - .boolean('R') - .alias('R', 'replacement-file') - .describe('R', - `Replacement is a filename from where the replacement will be generated. ` + - `If more than one line is found in the file the final replacement will be defined by each line trimmed and having newlines removed followed by all other rules (like -€).` - ) - - */ - - /* // Ideas - - // https://github.com/eugeneware/replacestream - .integer('M') - .describe('M', "Maximum length of match. Set this value only if any single file of your ") - .alias('M', 'max-match-len') - .default('M', false) - - - - .boolean('G') - .describe('G', "filename/globas are filename(s) for files containing one filename/globs on each line to be search/replaced") - .alias('G', 'globs-file') - - .boolean('j') - .describe('j', "Pattern is javascript source that will return a string giving the pattern to use") - .alias('j', 'pattern-js') - - - .boolean('glob-js') - .describe('glob-js', "filename/globs are javascript source that will return a string with newline seperating each glob to work on") - - - - -S --stream-size - - -l --stream-line - - - - - -Upper Case Free Lower Case Free - a -C -D -F - g - i -J -K k -N n -O -P - r - t -U -W w -Y y -Z z - - - */ - .boolean('c') - .alias('c', 'status-compact') - .describe( - 'c', - 'As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with `--quiet-total`. Will incude extra info about the inner workings of the engine in combination with the `--verbose` flag.' - ) - - .boolean('status') - .describe( - 'status', - 'Same as `--status-compact`, but always starting with a `\\n` followed by pretty printed JSON.' - ) - - .boolean('d') - .describe( - 'd', - 'Print only the status report, with what went into the engine and what it did. Does not block the output of --output and --output-match.' - ) - .alias('d', 'debug') - - .help('h') - .describe('h', 'Display help.') - .alias('h', 'help') - - .epilog(`Inspiration: .oO(What should 'sed' have been by now?)`); + ': Regexp search and replace for files using lookahead and backreference to matching groups in the replacement. Defaults to global multiline case-insensitive search.', + ['Usage', '> rexreplace pattern replacement [fileGlob|option]+'], + [ + 'Examples', + ["> rexreplace 'Foo' 'xxx' myfile.md", "'foobar' in myfile.md will become 'xxxbar'"], + '', + ['> rr xxx Foo myfile.md', "The alias 'rr' can be used instead of 'rexreplace'"], + '', + [ + "> rexreplace '(f?(o))o(.*)' '$3$1€2' myfile.md", + "'foobar' in myfile.md will become 'barfoo'", + ], + '', + [ + "> rexreplace '^#' '##' *.md", + 'All markdown files in this dir got all headlines moved one level deeper', + ], + '', + [ + "> rexreplace 'a' 'b' 'myfile.md' 'src/**/*.*' ", + 'Provide multiple files or glob if needed', + ], + ], + ], + outro: "Inspiration: .oO(What should 'sed' have been by now?)", + version: { + type: 'boolean', + alias: ['v'], + description: 'Print rexreplace version (can be given as only argument)', + }, + verbose: {type: 'boolean', alias: ['V'], description: 'More chatty output'}, + literal: { + type: 'boolean', + alias: ['L'], + description: 'Literal string search (no regex used when searching)', + }, + literalReplacement: { + type: 'boolean', + alias: ['l'], + description: + 'No "smart" values in the replacement, so you can have text like `$1` and `$&` without escaping them.', + }, + replacementFile: { + type: 'boolean', + description: + 'The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path.', + }, + trimFile: {type: 'boolean', description: 'Trim the content of the replacement file.'}, + trim: {type: 'boolean', description: 'Shorthand for --trim-pipe and --trim-file.'}, + voidIgnoreCase: { + type: 'boolean', + alias: ['I'], + description: 'Void case insensitive search. Applies to the pattern and to all globs.', + }, + voidGlobal: { + type: 'boolean', + alias: ['G'], + description: 'Void global search (stop looking after the first match).', + }, + dotAll: {type: 'boolean', alias: ['s'], description: 'Have `.` also match newline.'}, + voidMultiline: { + type: 'boolean', + alias: ['M'], + description: + 'Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line.', + }, + unicode: { + type: 'boolean', + alias: ['u'], + description: + 'Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode.', + }, + voidUnicode: { + type: 'boolean', + alias: ['U'], + description: + 'Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode.', + }, + flags: { + type: 'string', + description: + 'The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U.', + conflict: ['I', 'G', 'M', 's', 'u', 'U'], + }, + encoding: { + default: 'utf8', + alias: ['e'], + description: 'Encoding of files/piped data.', + type: 'string', + validate: value => Buffer.isEncoding(value) || `Unknown encoding: ${value}`, + }, + excludeRe: { + type: 'string[]', + alias: ['x'], + description: + 'Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times.', + }, + excludeGlob: { + type: 'string[]', + alias: ['X'], + description: + 'Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate.', + }, + engine: { + valid: [['rr', 'default'], 'regex+', 'runtime'], + alias: ['E'], + description: + 'The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun.', + }, + quiet: { + type: 'boolean', + alias: ['q'], + description: + 'Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown.', + }, + quietTotal: {type: 'boolean', alias: ['Q'], description: 'Never display errors or info'}, + fullPath: { + type: 'boolean', + alias: ['f'], + description: 'Print the full path of the named files instead of the relative path.', + }, + nameAll: { + type: 'boolean', + description: + 'Print the names of all files found by the globs, also the ones where nothing was changed.', + }, + nameDelimiter: { + type: 'string', + description: + 'String printed between the names of the files. Default is a line break, unless the name already ends with one.', + }, + contentDelimiter: { + type: 'string', + description: + 'String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one.', + }, + delimiter: { + type: 'string', + description: + 'Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\\t`, `\\n` and the `€` and `§` aliases.', + }, + nullOut: { + type: 'boolean', + alias: ['null'], + description: + "Shorthand for --delimiter '\\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out.", + }, + nameDelimiterTail: { + type: 'boolean', + description: + 'Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name.', + }, + contentDelimiterTail: { + type: 'boolean', + description: + 'Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe.', + }, + tail: { + type: 'boolean', + description: + 'Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail.', + }, + nl: { + type: 'boolean', + description: + 'Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed.', + }, + halt: {type: 'boolean', default: false, alias: ['H', 'bail'], description: 'Halt on first error'}, + voidEuro: { + type: 'boolean', + alias: ['€'], + description: + "Void having '€' as alias for '$'. The alias is used in all inputs, but never in globs, file names or piped data.", + }, + voidSection: { + type: 'boolean', + alias: ['§'], + description: + "Void having '§' as alias for '\\'. The alias is used in all inputs, but never in globs, file names or piped data.", + }, + output: { + type: 'boolean', + alias: ['o'], + description: + 'Output the final result instead of saving to file. Will also output content even if no replacement has taken place.', + }, + voidAsync: { + type: 'boolean', + alias: ['A'], + description: + 'Handle files in a synchronous flow. Good to limit memory usage when handling large files. ', + }, + voidBackup: { + type: 'boolean', + alias: ['B'], + description: + 'Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted.', + }, + keepBackup: { + type: 'boolean', + alias: ['b'], + description: 'Keep a backup file of the original content. With --rename the old file is kept.', + }, + simulate: { + type: 'boolean', + alias: ['S'], + description: + 'Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate.', + }, + sort: { + type: 'string', + valid: ['natural', 'ascii', 'modified', 'created', 'size', 'none'], + description: + 'Process files sync in a sorted order chosen from 6 values: `natural` (numbers compared as numbers and case insensitive), `ascii` (by char code, so `B` comes before `a`), `modified`, `created`, `size`, or `none` (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order.', + transform: value => String(value).toLowerCase(), + }, + sortReverse: {type: 'boolean', description: 'Turn the order of the files around.'}, + expectFiles: { + type: 'boolean', + alias: ['F'], + description: 'Check that at least one file is found. The exit code is 1 if not.', + }, + expectDiff: { + type: 'boolean', + alias: ['z'], + description: 'Check that at least one file is changed. The exit code is 1 if not.', + }, + expectZeroDiff: { + type: 'boolean', + alias: ['Z'], + description: 'Check that no file is changed. The exit code is 1 if one is.', + conflict: ['z', 'expect-diff-all'], + }, + expectDiffAll: { + type: 'boolean', + description: 'Check that all files found are changed. The exit code is 1 if not.', + }, + rename: { + type: 'boolean', + alias: ['r'], + description: + 'Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots.', + conflict: ['rename-path'], + }, + renamePath: { + type: 'boolean', + description: + 'Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made.', + }, + renameForce: { + type: 'boolean', + description: 'With --rename or --rename-path, write over a file that is there already.', + }, + outputMatch: { + type: 'boolean', + alias: ['m'], + description: + 'Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. The replacement is not used, so --js code is not run. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). ', + }, + trimPipe: { + type: 'boolean', + alias: ['T'], + description: + 'Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. ', + }, + replacementPipe: { + type: 'boolean', + alias: ['R'], + description: + 'Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter.', + }, + jsPipe: { + type: 'boolean', + description: 'Treat the replacement as javascript, and give the code the piped data as `pipe`.', + }, + globPipe: { + type: 'boolean', + alias: ['g'], + description: + 'Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments.', + conflict: ['R', 'js-pipe'], + }, + nullIn: { + type: 'boolean', + description: + 'The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out.', + demand: ['glob-pipe'], + }, + replacementJs: { + type: 'boolean', + alias: ['j', 'js'], + description: + "Treat replacement as javascript source code. \nThe statement from the last expression will become the replacement string. \nPurposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. \nThe full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. \nAt some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. \n\nThe code has access to the following variables: \n`r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, \n`fs` from node, \n`path` from node, \n`globs` from npm, \n`pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), \n`find`: pattern searched for (the needle), \n`text`: full text being searched i.e. file content or piped data (the haystack), \n`bytes`: total size of the haystack in bytes, \n`size`: human-friendly representation of the total size of the haystack, \n`time`: String representing the local time when the command was invoked,\n`time_obj`: date object representing `time`,\n`now`: alias for `time`,\n`cwd`: current process working dir, \n`nl`: a new-line char,\n`_`: a single space char (for easy string concatenation).\n\nThe following values defaults to `❌` if haystack does not originate from a file:\n`file`: contains the full path of the active file being searched (including full filename), \n`file_rel`: contains `file` relative to current process working dir, \n`file_path` and `file_path_rel`: the same as `file` and `file_rel`, \n`dirpath`: contains the full path without filename of the active file being searched, \n`dirpath_rel`: contains `dirpath` relative to current process working dir, \n`filename`: is the full filename of the active file being searched without path, \n`name`: filename of the active file being searched with no extension, \n`ext`: extension of the filename including leading dot, \n`mtime`: ISO inspired representation of the last local modification time of the current file, \n`ctime`: ISO representation of the local creation time of the current file. \n`mtime_obj`: date object representing `mtime`, \n`ctime_obj`: date object representing `ctime`.\n\nThe numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file, at its first match, unless it names the match, a group, `$m`, `$M` or `rr`. A file without a match does not run the code.\n\nAccess a named group as `$` and the name: `(?...)` gives you `$year`. A group named like a number, like `n`, wins over the number. With the flag `n` of `-E regex+` only named groups capture, so the code has no `$1`, while a replacement text still reads `$1` as the first named group, as in javascript. \n\n`rr` has the values with long names: `filePath`, `filePathRel`, `dirPath`, `dirPathRel`, `dirName`, `fileName`, `name`, `ext`, `bytes`, `size`, `mtime`, `ctime`, `mtimeObj`, `ctimeObj`, `fileNumber`, `filesTotal`, `matchedFileNumber`, `changedFileNumber`, `matchThisFile`, `matchUntilNow`, `text`, `find`, `stdin`, `cwd`, `time` and `timeObj`, and the groups, like `rr[1]` and `rr.year`, which overwrite a name they share. Code that names `rr`, also in a string or a comment, runs once per match. `print()` and `printErr()`, also on `rr`, print to stdout and stderr, also with --quiet-total. `pad(value, width = 3, char = '0')` pads a value, like `pad($1)` for `007`.\n\nAll variables, except from module, date objects, functions, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). \n", + }, + statusCompact: { + type: 'boolean', + alias: ['c'], + description: + 'As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with `--quiet-total`. Will incude extra info about the inner workings of the engine in combination with the `--verbose` flag.', + }, + status: { + type: 'boolean', + description: + 'Same as `--status-compact`, but always starting with a `\\n` followed by pretty printed JSON.', + }, + debug: { + type: 'boolean', + alias: ['d'], + description: + 'Print only the status report, with what went into the engine and what it did. Does not block the output of --output and --output-match.', + }, + help: {type: 'boolean', alias: ['h'], description: 'Display help.'}, +}; // The engine resolves with the status of the run, or throws a panic when the run can not go on. // The CLI is the only part that acts on the process. @@ -568,8 +367,9 @@ function exitOnPanic(config, e) { process.exit(3); } +// Help that was asked for goes to stdout, and the help with an error to stderr function backOut(exitcode = 1) { - yargs.showHelp(); + (0 === exitcode ? console.log : console.error)(argInfo({}, argConfig)); process.exit(exitcode); } @@ -596,15 +396,33 @@ if (0 < needHelp) { backOut(1 === needHelp ? 0 : 3); } +// Wrong flags are a panic, so the exit code is 3. The help and the message go to stderr. +const argv = argMate(process.argv.slice(2), argConfig, { + allowUnknown: false, + // A file can be named with only digits, like `2024`, so each glob is kept as the string that was given + allowAutoNumbers: false, + error: e => { + console.error(argInfo()); + console.error(e.message); + process.exit(3); + }, +}); + // All options into one big config object for the rexreplace core const config: any = {}; // Use only camelCase full length version of settings so we make sure the core can be documented properly -Object.keys(yargs.argv).forEach(key => { - if (1 < key.length && key.indexOf('-') < 0) { - config[key] = yargs.argv[key]; - } -}); +Object.assign(config, argv); +delete config._; + +if (config.help) { + backOut(0); +} + +if (config.version) { + console.log(version); + process.exit(0); +} if (config.trim) { config.trimPipe = config.trimFile = true; @@ -624,11 +442,11 @@ config.pipedData = null; config.pattern = pattern; -config.includeGlob = yargs.argv._; +config.includeGlob = argv._; -config.excludeGlob = [].concat(yargs.argv.excludeGlob).filter(Boolean); +config.excludeGlob = argv.excludeGlob ?? []; -config.excludeRe = [].concat(yargs.argv.excludeRe).filter(Boolean); +config.excludeRe = argv.excludeRe ?? []; config.replacement = replacement; diff --git a/test/cli/glob-pipe.test.js b/test/cli/glob-pipe.test.js index 67198606..19bf258a 100644 --- a/test/cli/glob-pipe.test.js +++ b/test/cli/glob-pipe.test.js @@ -20,7 +20,7 @@ describe('--glob-pipe', () => { async flag => { fs.writeFileSync('my.file', 'foobar\n'); const result = await sh($`echo -n my.file | bun ${cli} b x --glob-pipe ${flag}`); - expect(result.err).toContain('are mutually exclusive'); + expect(result.err).toContain('conflicts with'); expect(result.exit).toBe(3); expect(fs.readFileSync('my.file', 'utf8')).toBe('foobar\n'); } diff --git a/test/cli/pipe.test.js b/test/cli/pipe.test.js index 0e715b37..14dd6e94 100644 --- a/test/cli/pipe.test.js +++ b/test/cli/pipe.test.js @@ -29,6 +29,6 @@ describe('--js-pipe', () => { fs.writeFileSync('my.file', 'foobar\n'); const result = await sh($`echo -n x | bun ${cli} foo pipe my.file -o --js ${flag}`); expect(result.exit).toBe(3); - expect(result.err).toContain('Unknown argument'); + expect(result.err).toContain('Unknown parameter'); }); }); diff --git a/test/fix/783.test.js b/test/fix/783.test.js index de4abf5f..a04a1146 100644 --- a/test/fix/783.test.js +++ b/test/fix/783.test.js @@ -35,7 +35,7 @@ describe('#783 the exit code tells what happened', () => { const result = await sh($`bun ${cli} o x my.file --nope`); expect(result.exit).toBe(3); expect(result.out).toBe(''); - expect(result.err).toEndWith('Unknown argument: nope\n'); + expect(result.err).toEndWith("Unknown parameter 'nope' not allowed\n"); expect((await sh($`bun ${cli} o`)).exit).toBe(3); expect(fs.readFileSync('my.file', 'utf8')).toBe('foobar'); }); diff --git a/test/fix/854.test.js b/test/fix/854.test.js index f3d0700b..460f0ab6 100644 --- a/test/fix/854.test.js +++ b/test/fix/854.test.js @@ -211,7 +211,7 @@ describe('#854 --flags', () => { fs.writeFileSync('a.txt', 'a'); const result = await sh($`bun ${cli} a b a.txt --flags g -I`); expect(result.exit).toBe(3); - expect(result.err).toEndWith('Arguments flags and I are mutually exclusive\n'); + expect(result.err).toEndWith("The parameter '--flags' conflicts with '-I'\n"); expect(fs.readFileSync('a.txt', 'utf8')).toBe('a'); }); From a7c16105e84e53047cf00f42f99f1b8b5546f1a6 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Sat, 31 Oct 2026 22:48:36 +1100 Subject: [PATCH 4/9] Gather the performance tests in test/performance, one folder each, run with hyperfine --- test/performance/README.md | 33 ++++ .../tools/1984.html} | 0 test/performance/tools/run.sh | 72 ++++++++ test/performance/yargs-vs-argmate/run.sh | 49 ++++++ test/speed/run.sh | 11 -- test/speed/speed-code-test | 138 --------------- test/speed/speed-code-test-v2 | 138 --------------- test/speed/speed-code-test-v3 | 159 ------------------ test/speed/speed-test | 155 ----------------- test/speed/tesopi.js | 14 -- test/speed/testlog.speed.md | 110 ------------ test/speed/unicode.js | 14 -- 12 files changed, 154 insertions(+), 739 deletions(-) create mode 100644 test/performance/README.md rename test/{speed/1984.surge.sh.html => performance/tools/1984.html} (100%) create mode 100755 test/performance/tools/run.sh create mode 100755 test/performance/yargs-vs-argmate/run.sh delete mode 100755 test/speed/run.sh delete mode 100755 test/speed/speed-code-test delete mode 100755 test/speed/speed-code-test-v2 delete mode 100755 test/speed/speed-code-test-v3 delete mode 100755 test/speed/speed-test delete mode 100644 test/speed/tesopi.js delete mode 100644 test/speed/testlog.speed.md delete mode 100644 test/speed/unicode.js diff --git a/test/performance/README.md b/test/performance/README.md new file mode 100644 index 00000000..791cf48a --- /dev/null +++ b/test/performance/README.md @@ -0,0 +1,33 @@ +# Performance tests + +These tests measure how fast RexReplace is. They are not part of `yarn test`, because the times depend on the machine and on what else it is doing. Run them by hand, and compare numbers from the same machine only. + +Each test has its own folder with a `run.sh`. It builds rr from the working tree into a temp folder, runs [hyperfine](https://github.com/sharkdp/hyperfine) and prints the results. The runs start from the repo root, a usual place to run rr. + +## tools + +Compares rr with other tools that replace in files. Each tool removes the HTML tags of the book 1984 by George Orwell (`1984.html`) in place, on files of 1 KB, 10 KB, 100 KB, 1 MB, 10 MB and 100 MB. + +```bash +bash test/performance/tools/run.sh +bash test/performance/tools/run.sh 1KB 10MB +``` + +The tools are rr on Node, rr on Bun, `sed`, `gsed`, `perl`, `sd` and `replace-in-file`. A tool that is not installed is left out. + +`sed` and `perl` read a line at a time, so a tag that goes over two lines is not removed. rr and the other tools read the whole file, so their pattern stops at a line break. That way all the tools do the same work, and the files end up the same. + +A small file shows the start of each tool, and a big file shows the regex engine and the reading and writing of the file. + +## yargs-vs-argmate + +Compares the start of the CLI with yargs against the CLI of the working tree with argmate, on Node and on Bun. Most of a short run is the start, so this is where the parser of the flags shows. + +```bash +bash test/performance/yargs-vs-argmate/run.sh +bash test/performance/yargs-vs-argmate/run.sh main +``` + +The yargs side is built from the last commit where `src/cli.ts` used yargs, or from the git ref given as the first argument. Each side runs with no flags, with many flags and for the help. A run uses `-S`, so no file is changed. + +yargs reads every folder above the working folder to find a `package.json`. So under a big folder yargs is slower. In the temp folder of macOS, with tens of thousands of entries, it adds some 45 ms to each run. diff --git a/test/speed/1984.surge.sh.html b/test/performance/tools/1984.html similarity index 100% rename from test/speed/1984.surge.sh.html rename to test/performance/tools/1984.html diff --git a/test/performance/tools/run.sh b/test/performance/tools/run.sh new file mode 100755 index 00000000..f49d4762 --- /dev/null +++ b/test/performance/tools/run.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash + +# Compares rr from the working tree, on Node and on Bun, with other tools that replace in files. +# Each tool removes the HTML tags from a file in place, on sizes from 1 KB to 100 MB. +# A tool that is not installed is left out. Give sizes as arguments to run only those, like `1KB 10MB`. + +set -e + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$DIR/../../.." && pwd)" +SIZES="${*:-1KB 10KB 100KB 1MB 10MB 100MB}" + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +# The packages are external, so the builds sit next to a node_modules +ln -s "$ROOT/node_modules" "$TMP/node_modules" +(cd "$ROOT" && bun build src/cli.node.ts --target=node --format=esm --packages=external --outfile="$TMP/node.mjs" > /dev/null) +(cd "$ROOT" && bun build src/cli.bun.ts --target=bun --format=esm --packages=external --outfile="$TMP/bun.mjs" > /dev/null) + +# The book is 851 KB, so it is repeated to have enough text for the largest file +for i in $(seq 130); do cat "$DIR/1984.html"; done > "$TMP/book" + +# sed and perl read a line at a time, so a tag over two lines stays. +# The tools that read the whole file get a pattern that stops at a line break, to do the same work. +TAGS='<[^>]*>' +TAGS_IN_A_LINE='<[^>\n]*>' +FILE="$TMP/test.file" + +# BSD sed (macOS) needs an empty suffix after -i, GNU sed does not take one +if sed --version > /dev/null 2>&1; then + SED_INPLACE="-i" +else + SED_INPLACE="-i ''" +fi + +TOOLS=( + "rr on Node|node $TMP/node.mjs '$TAGS_IN_A_LINE' '' $FILE" + "rr on Bun|bun $TMP/bun.mjs '$TAGS_IN_A_LINE' '' $FILE" + "sed|sed $SED_INPLACE 's/$TAGS//g' $FILE" + "gsed|gsed -i 's/$TAGS//g' $FILE" + "perl|perl -pi -e 's/$TAGS//g' $FILE" + "sd|sd '$TAGS_IN_A_LINE' '' $FILE" + "replace-in-file|replace-in-file '/$TAGS_IN_A_LINE/g' '' $FILE --isRegex" +) + +ARGS=() +for tool in "${TOOLS[@]}"; do + name="${tool%%|*}" + command="${tool#*|}" + program="${command%% *}" + command -v "$program" > /dev/null || continue + ARGS+=(-n "$name" "$command") +done + +# The runs start from the repo, a usual place for rr +cd "$ROOT" + +for size in $SIZES; do + number="${size//[!0-9]/}" + unit="${size//[0-9]/}" + case "$unit" in + KB) bytes=$((number * 1024)) ;; + MB) bytes=$((number * 1024 * 1024)) ;; + *) echo "Unknown size $size, use KB or MB like 10KB" && exit 1 ;; + esac + head -c "$bytes" "$TMP/book" > "$TMP/$size.file" + + echo '' + echo "## $size" + hyperfine -N --warmup 2 --min-runs 10 --prepare "cp $TMP/$size.file $FILE" "${ARGS[@]}" +done diff --git a/test/performance/yargs-vs-argmate/run.sh b/test/performance/yargs-vs-argmate/run.sh new file mode 100755 index 00000000..3fdbd006 --- /dev/null +++ b/test/performance/yargs-vs-argmate/run.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash + +# Compares the start of the CLI with yargs against the CLI of the working tree, on Node and on Bun. +# Most of a short run is the start, so this is where the parser of the flags shows. +# Give a git ref as the first argument to compare against another commit. + +set -e + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "$ROOT" + +# The last commit with yargs is the parent of the one that took it out of src/cli.ts +BEFORE="${1:-$(git log -1 --format=%h -S"'yargs'" -- src/cli.ts)^}" + +TMP="$(mktemp -d)" +trap 'git -C "$ROOT" worktree remove --force "$TMP/yargs"; rm -rf "$TMP"' EXIT +git worktree add --quiet --detach "$TMP/yargs" "$BEFORE" +mkdir "$TMP/argmate" + +# The packages are external, so each build sits next to a node_modules. +# .mjs as the folders have no package.json that says the files are ES modules. +for side in yargs argmate; do + src="$ROOT" + [ "$side" = yargs ] && src="$TMP/yargs" + ln -sfn "$ROOT/node_modules" "$TMP/$side/node_modules" + (cd "$src" && bun build src/cli.node.ts --target=node --format=esm --packages=external --outfile="$TMP/$side/node.mjs" > /dev/null) + (cd "$src" && bun build src/cli.bun.ts --target=bun --format=esm --packages=external --outfile="$TMP/$side/bun.mjs" > /dev/null) +done + +echo 'foobar' > "$TMP/my.file" + +# The runs start from the repo, a usual place for rr. yargs reads each folder above the working folder, +# so in the temp folder of macOS, with tens of thousands of entries, yargs takes some 45 ms more. +cd "$ROOT" + +echo "yargs from $(git -C "$ROOT" log -1 --format='%h %s' "$BEFORE")" +echo 'argmate from the working tree' + +compare() { + hyperfine -N --warmup 10 --runs 40 \ + -n "yargs $1" "$2 $TMP/yargs/$3 $4" \ + -n "argmate $1" "$2 $TMP/argmate/$3 $4" +} + +for runtime in node bun; do + compare "$runtime, no flags" "$runtime" "$runtime.mjs" "o x $TMP/my.file -S" + compare "$runtime, many flags" "$runtime" "$runtime.mjs" "o x $TMP/my.file -S -V -m -e utf8 --sort natural -x nope -X nope --name-delimiter , --expect-files" + compare "$runtime, help" "$runtime" "$runtime.mjs" "-h" +done diff --git a/test/speed/run.sh b/test/speed/run.sh deleted file mode 100755 index b428f3ef..00000000 --- a/test/speed/run.sh +++ /dev/null @@ -1,11 +0,0 @@ - -DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - -# bash $DIR/speed-test ; exit # for debug - -echo Running tests -bash $DIR/speed-test &> $DIR/testlog.speed.md -cat $DIR/testlog.speed.md -echo '' -echo Tests stored in testlog.speed.md -open $DIR/testlog.speed.md || true diff --git a/test/speed/speed-code-test b/test/speed/speed-code-test deleted file mode 100755 index 7bf09f86..00000000 --- a/test/speed/speed-code-test +++ /dev/null @@ -1,138 +0,0 @@ -#!/bin/bash - -DATETIME=`date '+%Y-%m-%d %H:%M'` -RR_VERSION="v`rr -v`" -NODE_VERSION=`NODE -v` -echo '' -echo "# RexReplace vs \`sed\` speed test" -printf "\n\t%s\n\t%s\n\t%s\n\n" "$DATETIME" "RexReplace $RR_VERSION" "Node $NODE_VERSION" - -echo "This test will remove HTML tags from the HTML sorce of George Orwell's 1984 to get the pure text version." - -echo 'All testresults printed are the sum of 10 rounds given in seconds.' - -TIMEFORMAT=%R - -create_testdata(){ - #return # for debug - - curl -s 1984.surge.sh > source.file - head -c $((1*1024)) source.file > 1Kb.file - head -c $((5*1024)) source.file > 5Kb.file - head -c $((10*1024)) source.file > 10Kb.file - head -c $((100*1024)) source.file > 100Kb.file - head -c $((500*1024)) source.file > 500Kb.file - echo '' > 1Mb.file; for i in {1..2}; do cat 500Kb.file >> 1Mb.file; done - echo '' > 5Mb.file; for i in {1..5}; do cat 1Mb.file >> 5Mb.file; done - echo '' > 10Mb.file; for i in {1..10}; do cat 1Mb.file >> 10Mb.file; done - echo '' > 25Mb.file; for i in {1..25}; do cat 1Mb.file >> 25Mb.file; done - echo '' > 50Mb.file; for i in {1..50}; do cat 1Mb.file >> 50Mb.file; done - echo '' > 100Mb.file; for i in {1..100}; do cat 1Mb.file >> 100Mb.file; done - - -} - -remove_testdata(){ - #return # for debug - - rm source.file - rm 1Kb.file - rm 5Kb.file - rm 10Kb.file - rm 100Kb.file - # rm 500Kb.file - # rm 1Mb.file - # rm 5Mb.file - # rm 10Mb.file - # rm 25Mb.file - # rm 50Mb.file - # rm 100Mb.file - - - -} - - -run_rr(){ - - echo '' - printf " - RexReplace $2 on a %s file: \t" $1 - time for i in {1..100}; do - $2 '<.*?>' '' "$1.file" -o > /dev/null - done - echo '' - -} - - - -do_test(){ - - # Do tests - echo '' - echo "## A single $1 file" - echo '' - - echo '_100 rounds of:_' - echo '' - printf " - sed on %s file: \t" $1 - time for i in {1..100}; do - cat "$1.file" | sed 's/<.*?>//g' > /dev/null - done - - #run_rr $1 rr_3_m -# run_rr $1 rr_5_m - # run_rr $1 rr_6_m - #run_rr $1 rr_6_b - #run_rr $1 rr_6_b_u - #run_rr $1 rr_6_c - #run_rr $1 rr_6_c_js - #run_rr $1 rr_6 - #run_rr $1 rr_6_b_c -# run_rr $1 rr_6_b_c_js - #run_rr $1 rr_6_b_u_c - #run_rr $1 rr_6_b_u_c_js - - -# Total 1,977 98,141 94,509 95,242 96,773 95,887 95,728 97,053 94,747 94,314 95,418 96,743 -# Compination sed rr_3_m rr_5_m rr_6_b rr_6_b_u rr_6_c rr_6_c_js rr_6 rr_6_b_c rr_6_b_c_js rr_6_b_u_c rr_6_b_u_c_js -# 1kb 0,633 24,249 23,337 24,131 24,392 24,705 24,553 24,673 23,645 23,322 23,384 25,703 -# 5kb 0,498 26,377 23,582 23,448 25,37 23,79 23,734 24,218 23,792 23,895 23,559 23,768 -# 10kb 0,401 23,889 23,837 23,909 23,506 23,818 23,961 24,217 23,807 23,434 24,984 23,757 -# 100kb 0,445 23,626 23,753 23,754 23,505 23,574 23,48 23,945 23,503 23,663 23,491 23,515 - - -# run_rr $1 rr_5_b -# run_rr $1 rr_5_b_c_js - - - run_rr $1 rr - -} - - -create_testdata - -do_test '1Kb' -do_test '5Kb' -do_test '10Kb' -do_test '100Kb' -# do_test '500Kb' -# do_test '1Mb' -# do_test '5Mb' -# do_test '10Mb' -# do_test '25Mb' -# do_test '50Mb' -# do_test '100Mb' - - - - -echo '' -echo '----' -echo '' -echo "_All tests completed_" - -remove_testdata - - diff --git a/test/speed/speed-code-test-v2 b/test/speed/speed-code-test-v2 deleted file mode 100755 index 7a934bc7..00000000 --- a/test/speed/speed-code-test-v2 +++ /dev/null @@ -1,138 +0,0 @@ -#!/bin/bash - -DATETIME=`date '+%Y-%m-%d %H:%M'` -RR_VERSION="v`rr -v`" -NODE_VERSION=`NODE -v` -echo '' -echo "# RexReplace vs \`sed\` speed test" -printf "\n\t%s\n\t%s\n\t%s\n\n" "$DATETIME" "RexReplace $RR_VERSION" "Node $NODE_VERSION" - -echo "This test will remove HTML tags from the HTML sorce of George Orwell's 1984 to get the pure text version." - -echo 'All testresults printed are the sum of 10 rounds given in seconds.' - -TIMEFORMAT=%R - -create_testdata(){ - #return # for debug - - curl -s 1984.surge.sh > source.file - head -c $((1*1024)) source.file > 1Kb.file - head -c $((5*1024)) source.file > 5Kb.file - head -c $((10*1024)) source.file > 10Kb.file - head -c $((100*1024)) source.file > 100Kb.file - head -c $((500*1024)) source.file > 500Kb.file - echo '' > 1Mb.file; for i in {1..2}; do cat 500Kb.file >> 1Mb.file; done - echo '' > 5Mb.file; for i in {1..5}; do cat 1Mb.file >> 5Mb.file; done - echo '' > 10Mb.file; for i in {1..10}; do cat 1Mb.file >> 10Mb.file; done - echo '' > 25Mb.file; for i in {1..25}; do cat 1Mb.file >> 25Mb.file; done - echo '' > 50Mb.file; for i in {1..50}; do cat 1Mb.file >> 50Mb.file; done - echo '' > 100Mb.file; for i in {1..100}; do cat 1Mb.file >> 100Mb.file; done - - -} - -remove_testdata(){ - #return # for debug - - rm source.file - rm 1Kb.file - rm 5Kb.file - rm 10Kb.file - rm 100Kb.file - # rm 500Kb.file - # rm 1Mb.file - # rm 5Mb.file - # rm 10Mb.file - # rm 25Mb.file - # rm 50Mb.file - # rm 100Mb.file - - - -} - - -run_rr(){ - - echo '' - printf " - RexReplace $2 on a %s file: \t" $1 - time for i in {1..100}; do - $2 '<.*?>' '' "$1.file" -o > /dev/null - done - echo '' - -} - - - -do_test(){ - - # Do tests - echo '' - echo "## A single $1 file" - echo '' - - echo '_100 rounds of:_' - echo '' - printf " - sed on %s file: \t" $1 - time for i in {1..100}; do - cat "$1.file" | sed 's/<.*?>//g' > /dev/null - done - - #run_rr $1 rr_3_m -# run_rr $1 rr_5_m - # run_rr $1 rr_6_m - #run_rr $1 rr_6_b - #run_rr $1 rr_6_b_u - #run_rr $1 rr_6_c - #run_rr $1 rr_6_c_js - #run_rr $1 rr_6 - #run_rr $1 rr_6_b_c -# run_rr $1 rr_6_b_c_js - #run_rr $1 rr_6_b_u_c - #run_rr $1 rr_6_b_u_c_js - - -# Total 1,977 98,141 94,509 95,242 96,773 95,887 95,728 97,053 94,747 94,314 95,418 96,743 -# Compination sed rr_3_m rr_5_m rr_6_b rr_6_b_u rr_6_c rr_6_c_js rr_6 rr_6_b_c rr_6_b_c_js rr_6_b_u_c rr_6_b_u_c_js -# 1kb 0,633 24,249 23,337 24,131 24,392 24,705 24,553 24,673 23,645 23,322 23,384 25,703 -# 5kb 0,498 26,377 23,582 23,448 25,37 23,79 23,734 24,218 23,792 23,895 23,559 23,768 -# 10kb 0,401 23,889 23,837 23,909 23,506 23,818 23,961 24,217 23,807 23,434 24,984 23,757 -# 100kb 0,445 23,626 23,753 23,754 23,505 23,574 23,48 23,945 23,503 23,663 23,491 23,515 - - -# run_rr $1 rr_5_b -# run_rr $1 rr_5_b_c_js - - - run_rr $1 rr - -} - - -create_testdata - -do_test '1Kb' -do_test '5Kb' -do_test '10Kb' -do_test '100Kb' - do_test '500Kb' - do_test '1Mb' - do_test '5Mb' - do_test '10Mb' - do_test '25Mb' - do_test '50Mb' - do_test '100Mb' - - - - -echo '' -echo '----' -echo '' -echo "_All tests completed_" - -remove_testdata - - diff --git a/test/speed/speed-code-test-v3 b/test/speed/speed-code-test-v3 deleted file mode 100755 index aa904734..00000000 --- a/test/speed/speed-code-test-v3 +++ /dev/null @@ -1,159 +0,0 @@ -#!/bin/bash - -DATETIME=`date '+%Y-%m-%d %H:%M'` -RR_VERSION="v`rr -v`" -NODE_VERSION=`NODE -v` -DATAPOND="datapond.txt" -echo '' -echo "# RexReplace vs x speed test" -printf "\n\t%s\n\t%s\n\t%s\n\n" "$DATETIME" "RexReplace $RR_VERSION" "Node $NODE_VERSION" - -echo "This test will remove p and div HTML tags from the HTML sorce of George Orwell's 1984." - -echo 'All testresults printed are the sum of 10 rounds given in seconds.' - -TIMEFORMAT=%R - -create_testdata(){ - #return # for debug - - #curl -s 1984.surge.sh > source.file - cat 1984.surge.sh.html > source.file - tail -c $((1*1024)) source.file > 1Kb.file - tail -c $((5*1024)) source.file > 5Kb.file - tail -c $((10*1024)) source.file > 10Kb.file - tail -c $((100*1024)) source.file > 100Kb.file - tail -c $((500*1024)) source.file > 500Kb.file - echo '' > 1Mb.file; for i in {1..2}; do cat 500Kb.file >> 1Mb.file; done - echo '' > 5Mb.file; for i in {1..5}; do cat 1Mb.file >> 5Mb.file; done - echo '' > 10Mb.file; for i in {1..10}; do cat 1Mb.file >> 10Mb.file; done - echo '' > 25Mb.file; for i in {1..25}; do cat 1Mb.file >> 25Mb.file; done - echo '' > 50Mb.file; for i in {1..50}; do cat 1Mb.file >> 50Mb.file; done - echo '' > 100Mb.file; for i in {1..100}; do cat 1Mb.file >> 100Mb.file; done - - -} - -remove_testdata(){ - return # for debug - - rm source.file - rm 1Kb.file - rm 5Kb.file - rm 10Kb.file - rm 100Kb.file - rm 500Kb.file - rm 1Mb.file - rm 5Mb.file - rm 10Mb.file - rm 25Mb.file - rm 50Mb.file - rm 100Mb.file - - -} - - -run_hyperfine(){ - echo '' - printf "## Running test on %s file\n" $1 - echo '' - cp "$1.file" "test$1.file" - hyperfine \ - --time-unit second \ - --warmup 5 \ - --min-runs 10 \ - --prepare "cp $1.file test$1.file" \ - "sed -i '' 's:$2:$3:g' test$1.file" \ - "gsed -i -e 's/$2/$3/g' test$1.file" \ - "rr '$2' '$3' test$1.file" \ - "perl -pi -e 's/$2/$3/g' test$1.file" \ - "awk '{gsub(/$2/,\"$3\")}' test$1.file" \ - "replace-in-file '/$2/g' '$3' test$1.file --isRegex" \ - "replace '$2' '$3' test$1.file" \ - "replace-x '$2' '$3' test$1.file" \ - "sd -i '$2' '$3' test$1.file" \ - --export-csv output$1.csv \ - --export-json output$1.json \ - --export-markdown output$1.md \ - # --show-output - echo '' - - echo '----' -} - - -run_hyperfine_specific(){ - echo '' - printf "## Running test on %s file\n" $1 - echo '' - cp "$1.file" "test$1.file" - echo hyperfine \ - --time-unit second \ - --warmup 5 \ - --min-runs 10 \ - --prepare "cp $1.file test$1.file" \ - "sed -i '' 's:]*>:$3:g' test$1.file" \ - "gsed -i -e 's:$2:$3:g' test$1.file" \ - "rexreplace '$2' '$3' test$1.file" \ - "perl -pi -e 's:$2:$3:g' test$1.file" \ - "replace-in-file '/<\/*(p|span|div)[^>]*>/g' '$3' test$1.file --isRegex" \ - "replace '$2' '$3' test$1.file" \ - "replace-x '$2' '$3' test$1.file" \ - "sd -i '$2' '$3' test$1.file" \ - --export-csv output$1.csv \ - --export-json output$1.json \ - --export-markdown output$1.md \ - # --show-output - #"awk '{gsub(:$2:,\"$3\")}' test$1.file" \ - echo '' - - echo '----' - - echo '' >> "$DATAPOND" - - echo `stat -f%z "$1.file"` >> "$DATAPOND" - - alasql 'COLUMN OF SELECT mean from csv(?)' "output$1.csv" >> "$DATAPOND" -} - - - -do_test(){ - - #run_hyperfine_specific $1 ']*>' 'foobar' - run_hyperfine $1 'e' 'foobar' - -} - - -create_testdata - -echo '' > "$DATAPOND" - -do_test '1Kb' -do_test '5Kb' -do_test '10Kb' -do_test '100Kb' -do_test '500Kb' -do_test '1Mb' -do_test '5Mb' -do_test '10Mb' -do_test '25Mb' -do_test '50Mb' -do_test '100Mb' - - - - -echo '' -echo "_All tests completed_" - - -alasql 'COLUMN OF SELECT command from csv(?)' output1kb.csv >> "$DATAPOND" - -rr '\[|\]|^\s+"?|[, ].*$' '' "$DATAPOND" -Q - -remove_testdata - - diff --git a/test/speed/speed-test b/test/speed/speed-test deleted file mode 100755 index 9f5ff2fe..00000000 --- a/test/speed/speed-test +++ /dev/null @@ -1,155 +0,0 @@ -#!/bin/bash - -DATETIME=`date '+%Y-%m-%d %H:%M'` -RR_VERSION="v`rr -v`" -NODE_VERSION=`NODE -v` -echo '' -echo "# RexReplace vs \`sed\` speed test" -printf "\n\t%s\n\t%s\n\t%s\n\n" "$DATETIME" "RexReplace $RR_VERSION" "Node $NODE_VERSION" - -echo "This test will remove HTML tags from the HTML sorce of George Orwell's 1984 to get the pure text version." - -echo 'All testresults printed are the sum of 10 rounds given in seconds.' - -TIMEFORMAT=%R - -create_testdata(){ - #return # for debug - - curl -s 1984.surge.sh > source.file - head -c $((1*1024)) source.file > 1Kb.file - head -c $((5*1024)) source.file > 5Kb.file - head -c $((10*1024)) source.file > 10Kb.file - head -c $((100*1024)) source.file > 100Kb.file - head -c $((500*1024)) source.file > 500Kb.file - echo '' > 1Mb.file; for i in {1..2}; do cat 500Kb.file >> 1Mb.file; done - echo '' > 5Mb.file; for i in {1..5}; do cat 1Mb.file >> 5Mb.file; done - echo '' > 10Mb.file; for i in {1..10}; do cat 1Mb.file >> 10Mb.file; done - echo '' > 25Mb.file; for i in {1..25}; do cat 1Mb.file >> 25Mb.file; done - echo '' > 50Mb.file; for i in {1..50}; do cat 1Mb.file >> 50Mb.file; done - echo '' > 100Mb.file; for i in {1..100}; do cat 1Mb.file >> 100Mb.file; done -} - -remove_testdata(){ - #return # for debug - - rm source.file - rm 1Kb.file - rm 5Kb.file - rm 10Kb.file - rm 100Kb.file - rm 500Kb.file - rm 1Mb.file - rm 5Mb.file - rm 10Mb.file - rm 25Mb.file - rm 50Mb.file - rm 100Mb.file - - rm A0.txt - rm A1.txt - rm A2.txt - rm A3.txt - rm A4.txt - rm A5.txt - rm A6.txt - rm A7.txt - rm A8.txt - rm A9.txt - -} - -do_test(){ - - # Do tests - echo '' - echo "## A single $1 file" - echo '' - - echo '_10 rounds of:_' - echo '' - printf " - sed on %s file: \t" $1 - time for i in {1..10}; do - cat "$1.file" | sed 's/<.*?>//g' > /dev/null - done - - echo '' - printf " - RexReplace on a %s file: \t" $1 - time for i in {1..10}; do - rexreplace_ '<.*?>' '' "$1.file" -o > /dev/null - done - echo '' - -} - - -create_testdata - -do_test '1Kb' -do_test '5Kb' -do_test '10Kb' -do_test '100Kb' -do_test '500Kb' -do_test '1Mb' -do_test '5Mb' -do_test '10Mb' -do_test '25Mb' -do_test '50Mb' -do_test '100Mb' - - - -echo '' -echo '' -echo '----' -echo '' -echo '' -echo "## 10 files (10Kb each) - time for 10 rounds" -cat 10Kb.file > A0.txt -cat A0.txt > A1.txt -cat A0.txt > A2.txt -cat A0.txt > A3.txt -cat A0.txt > A4.txt -cat A0.txt > A5.txt -cat A0.txt > A6.txt -cat A0.txt > A7.txt -cat A0.txt > A8.txt -cat A0.txt > A9.txt - -echo '' -printf " - sed on 10 files: \t" -time for i in {1..10}; do - cat A0.txt | sed 's/<.*>//g' > /dev/null - cat A1.txt | sed 's/<.*>//g' > /dev/null - cat A2.txt | sed 's/<.*>//g' > /dev/null - cat A3.txt | sed 's/<.*>//g' > /dev/null - cat A4.txt | sed 's/<.*>//g' > /dev/null - cat A5.txt | sed 's/<.*>//g' > /dev/null - cat A6.txt | sed 's/<.*>//g' > /dev/null - cat A7.txt | sed 's/<.*>//g' > /dev/null - cat A8.txt | sed 's/<.*>//g' > /dev/null - cat A9.txt | sed 's/<.*>//g' > /dev/null -done - - -echo '' -printf " - RexReplace on 10 files: \t" -time for i in {1..10}; do - rexreplace '<.*>' '' -o A0.txt A1.txt A2.txt A3.txt A4.txt A5.txt A6.txt A7.txt A8.txt A9.txt > /dev/null -done - -echo '' -printf " - RexReplace on 10 files given as one glob: \t" -time for i in {1..10}; do - rexreplace '<.*>' '' -o A*.txt > /dev/null -done - - -echo '' -echo '----' -echo '' -echo "_All tests completed_" - -remove_testdata - - diff --git a/test/speed/tesopi.js b/test/speed/tesopi.js deleted file mode 100644 index faab71c9..00000000 --- a/test/speed/tesopi.js +++ /dev/null @@ -1,14 +0,0 @@ -describe(__filename.replace(/.*\/([^\/]+)\.js$/, '$1') + ' - Stub', function () { - before(function () {}); - - after(function () {}); - - it('rr is awesome', function () { - rexreplace({ - pattern: 'a', - replacement: 'b', - files: ['myfile'], - }); - assert.equal(1, 1); - }); -}); diff --git a/test/speed/testlog.speed.md b/test/speed/testlog.speed.md deleted file mode 100644 index 9c07f966..00000000 --- a/test/speed/testlog.speed.md +++ /dev/null @@ -1,110 +0,0 @@ -# RexReplace vs `sed` speed test - - 2018-07-07 00:10 - RexReplace v3.1.0 - Node v8.11.3 - -This test will remove HTML tags from the HTML sorce of George Orwell's 1984 to get the pure text version. -All testresults printed are the sum of 10 rounds given in seconds. - -## A single 1Kb file - -_10 rounds of:_ - -- sed on 1Kb file: 0.140 - -- RexReplace on a 1Kb file: 5.482 - -## A single 5Kb file - -_10 rounds of:_ - -- sed on 5Kb file: 0.139 - -- RexReplace on a 5Kb file: 4.509 - -## A single 10Kb file - -_10 rounds of:_ - -- sed on 10Kb file: 0.101 - -- RexReplace on a 10Kb file: 4.213 - -## A single 100Kb file - -_10 rounds of:_ - -- sed on 100Kb file: 0.167 - -- RexReplace on a 100Kb file: 3.213 - -## A single 500Kb file - -_10 rounds of:_ - -- sed on 500Kb file: 0.458 - -- RexReplace on a 500Kb file: 3.188 - -## A single 1Mb file - -_10 rounds of:_ - -- sed on 1Mb file: 0.913 - -- RexReplace on a 1Mb file: 3.570 - -## A single 5Mb file - -_10 rounds of:_ - -- sed on 5Mb file: 4.819 - -- RexReplace on a 5Mb file: 4.362 - -## A single 10Mb file - -_10 rounds of:_ - -- sed on 10Mb file: 8.433 - -- RexReplace on a 10Mb file: 5.297 - -## A single 25Mb file - -_10 rounds of:_ - -- sed on 25Mb file: 17.726 - -- RexReplace on a 25Mb file: 7.144 - -## A single 50Mb file - -_10 rounds of:_ - -- sed on 50Mb file: 44.416 - -- RexReplace on a 50Mb file: 13.570 - -## A single 100Mb file - -_10 rounds of:_ - -- sed on 100Mb file: 72.585 - -- RexReplace on a 100Mb file: 23.618 - ---- - -## 10 files (10Kb each) - time for 10 rounds - -- sed on 10 files: 0.778 - -- RexReplace on 10 files: 3.015 - -- RexReplace on 10 files given as one glob: 3.313 - ---- - -_All tests completed_ diff --git a/test/speed/unicode.js b/test/speed/unicode.js deleted file mode 100644 index faab71c9..00000000 --- a/test/speed/unicode.js +++ /dev/null @@ -1,14 +0,0 @@ -describe(__filename.replace(/.*\/([^\/]+)\.js$/, '$1') + ' - Stub', function () { - before(function () {}); - - after(function () {}); - - it('rr is awesome', function () { - rexreplace({ - pattern: 'a', - replacement: 'b', - files: ['myfile'], - }); - assert.equal(1, 1); - }); -}); From 8719b08aa6c4d8c8237af31481862d43d21acdbf Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Sat, 31 Oct 2026 22:48:36 +1100 Subject: [PATCH 5/9] Keep the yargs comments, drop -help and -version, and build the README options from src/options.ts --- README.md | 124 +++++++------ npm/load-options.ts | 28 +++ package.json | 3 +- src/cli.ts | 333 +--------------------------------- src/options.ts | 422 +++++++++++++++++++++++++++++++++++++++++++ test/cli/run.sh | 7 + test/fix/692.test.js | 20 +- 7 files changed, 541 insertions(+), 396 deletions(-) create mode 100644 npm/load-options.ts create mode 100644 src/options.ts diff --git a/README.md b/README.md index 8007ad0a..ffebe7f5 100644 --- a/README.md +++ b/README.md @@ -311,67 +311,66 @@ If planned renames clash, the process is halted and no change is applied to any > rexreplace pattern replacement [fileGlob|option]+ ``` -| Flag | Effect | -| ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `-v` | **`--version`** Print rexreplace version (can be given as only argument) [boolean] | -| `-V` | **`--verbose`** More chatty output [boolean] | -| `-L` | **`--literal`** Literal string search (no regex used when searching) [boolean] | -| `-l` | **`--literal-replacement`** No "smart" values in the replacement, so you can have text like `$1` and `$&` without escaping them. [boolean] | -| | **`--replacement-file`** The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path. [boolean] | -| `-I` | **`--void-ignore-case`** Void case insensitive search. Applies to the pattern and to all globs. [boolean] | -| `-G` | **`--void-global`** Void global search (stop looking after the first match). [boolean] | -| `-s` | **`--dot-all`** Have `.` also match newline. [boolean] | -| `-M` | **`--void-multiline`** Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] | -| `-u` | **`--unicode`** Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode. [boolean] | -| `-U` | **`--void-unicode`** Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode. [boolean] | -| | **`--flags`** The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U. [string] | -| `-e` | **`--encoding`** Encoding of files/piped data. [default: "utf8"] | -| `-E` | **`--engine`** The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun. [choices: "rr", "default", "regex+", "runtime"] [default: "rr"] | -| `-q` | **`--quiet`** Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown. [boolean] | -| `-Q` | **`--quiet-total`** Never display errors or info [boolean] | -| `-f` | **`--full-path`** Print the full path of the named files instead of the relative path. [boolean] | -| | **`--name-all`** Print the names of all files found by the globs, also the ones where nothing was changed. [boolean] | -| | **`--name-delimiter`** String printed between the names of the files. Default is a line break, unless the name already ends with one. [string] | -| | **`--content-delimiter`** String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one. [string] | -| | **`--delimiter`** Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\t`, `\n` and the `€` and `§` aliases. [string] | -| | **`--null-out`**, **`--null`** Shorthand for --delimiter '\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out. [boolean] | -| | **`--name-delimiter-tail`** Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name. [boolean] | -| | **`--content-delimiter-tail`** Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe. [boolean] | -| | **`--tail`** Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail. [boolean] | -| | **`--nl`** Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed. [boolean] | -| `-H` | **`--halt`**, **`--bail`** Halt on first error [boolean] [default: false] | -| `-c` | **`--status-compact`** As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with `--quiet-total`. Will incude extra info about the inner workings of the engine in combination with the `--verbose` flag. [boolean] | -| | **`--status`** Same as `--status-compact`, but always starting with a `\n` followed by pretty printed JSON. [boolean] | -| `-d` | **`--debug`** Print only the status report, with what went into the engine and what it did. Does not block the output of --output and --output-match. [boolean] | - -| `-€` | **`--void-euro`** Void having `€` as alias for `$`. The alias is used in all inputs, but never in globs, file names or piped data. [boolean] | -| `-§` | **`--void-section`** Void having `§` as alias for `\`. The alias is used in all inputs, but never in globs, file names or piped data. [boolean] | -| `-o` | **`--output`** Output the final result instead of saving to file. Will also output content even if no replacement has taken place. [boolean] | -| `-A` | **`--void-async`** Handle files in a synchronous flow. Good to limit memory usage when handling large files. [boolean] | -| `-B` | **`--void-backup`** Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted. [boolean] | -| `-b` | **`--keep-backup`** Keep a backup file of the original content. With --rename the old file is kept. [boolean] | -| `-S` | **`--simulate`** Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate. [boolean] | -| | **`--sort`** Process files sync in a sorted order chosen from 6 values: `natural` (numbers compared as numbers and case insensitive), `ascii` (by char code, so `B` comes before `a`), `modified`, `created`, `size`, or `none` (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order. [string] [choices: "natural", "ascii", "modified", "created", "size", "none"] | -| | **`--sort-reverse`** Turn the order of the files around. [boolean] | -| `-F` | **`--expect-files`** Check that at least one file is found. The exit code is 1 if not. [boolean] | -| `-z` | **`--expect-diff`** Check that at least one file is changed. The exit code is 1 if not. [boolean] | -| `-Z` | **`--expect-zero-diff`** Check that no file is changed. The exit code is 1 if one is. [boolean] | -| | **`--expect-diff-all`** Check that all files found are changed. The exit code is 1 if not. [boolean] | -| `-r` | **`--rename`** Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots. [boolean] | -| | **`--rename-path`** Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made. [boolean] | -| | **`--rename-force`** With --rename or --rename-path, write over a file that is there already. [boolean] | -| `-m` | **`--output-match`** Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. The replacement is not used, so --js code is not run. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). [boolean] | -| `-T` | **`--trim-pipe`** Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean] | -| | **`--trim-file`** Trim the content of the replacement file. [boolean] | -| | **`--trim`** Shorthand for --trim-pipe and --trim-file. [boolean] | -| | **`--js-pipe`** Treat the replacement as javascript, and give the code the piped data as `pipe`. [boolean] | -| `-g` | **`--glob-pipe`** Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] | -| | **`--null-in`** The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out. [boolean] | -| `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | -| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `file_path` and `file_path_rel`: the same as `file` and `file_rel`, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file, at its first match, unless it names the match, a group, `$m`, `$M` or `rr`. A file without a match does not run the code. Access a named group as `$` and the name: `(?...)` gives you `$year`. A group named like a number, like `n`, wins over the number. With the flag `n` of `-E regex+` only named groups capture, so the code has no `$1`, while a replacement text still reads `$1` as the first named group, as in javascript. `rr` has the values with long names: `filePath`, `filePathRel`, `dirPath`, `dirPathRel`, `dirName`, `fileName`, `name`, `ext`, `bytes`, `size`, `mtime`, `ctime`, `mtimeObj`, `ctimeObj`, `fileNumber`, `filesTotal`, `matchedFileNumber`, `changedFileNumber`, `matchThisFile`, `matchUntilNow`, `text`, `find`, `stdin`, `cwd`, `time` and `timeObj`, and the groups, like `rr[1]` and `rr.year`, which overwrite a name they share. Code that names `rr`, also in a string or a comment, runs once per match. `print()` and `printErr()`, also on `rr`, print to stdout and stderr, also with --quiet-total. `pad(value, width = 3, char = '0')` pads a value, like `pad($1)` for `007`. All variables, except from module, date objects, functions, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | -| `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times. [string] | -| `-X` | **`--exclude-glob`** Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate. [string] | -| `-h` | **`--help`** Display help. [boolean] | +| Short | Long | Description | +| --- | --- | --- | +| `-v` | `--version` | Print rexreplace version (can be given as only argument) [boolean] | +| `-V` | `--verbose` | More chatty output [boolean] | +| `-L` | `--literal` | Literal string search (no regex used when searching) [boolean] | +| `-l` | `--literal-replacement` | No "smart" values in the replacement, so you can have text like `$1` and `$&` without escaping them. [boolean] | +| | `--replacement-file` | The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path. [boolean] | +| | `--trim-file` | Trim the content of the replacement file. [boolean] | +| | `--trim` | Shorthand for --trim-pipe and --trim-file. [boolean] | +| `-I` | `--void-ignore-case` | Void case insensitive search. Applies to the pattern and to all globs. [boolean] | +| `-G` | `--void-global` | Void global search (stop looking after the first match). [boolean] | +| `-s` | `--dot-all` | Have `.` also match newline. [boolean] | +| `-M` | `--void-multiline` | Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] | +| `-u` | `--unicode` | Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode. [boolean] | +| `-U` | `--void-unicode` | Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode. [boolean] | +| | `--flags` | The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U. [string] | +| `-e` | `--encoding` | Encoding of files/piped data. [string=utf8] | +| `-x` | `--exclude-re` | Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times. [string[]] | +| `-X` | `--exclude-glob` | Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate. [string[]] | +| `-E` | `--engine` | The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun. [rr/default, regex+, runtime] =rr | +| `-q` | `--quiet` | Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown. [boolean] | +| `-Q` | `--quiet-total` | Never display errors or info [boolean] | +| `-f` | `--full-path` | Print the full path of the named files instead of the relative path. [boolean] | +| | `--name-all` | Print the names of all files found by the globs, also the ones where nothing was changed. [boolean] | +| | `--name-delimiter` | String printed between the names of the files. Default is a line break, unless the name already ends with one. [string] | +| | `--content-delimiter` | String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one. [string] | +| | `--delimiter` | Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\t`, `\n` and the `€` and `§` aliases. [string] | +| | `--null-out`, `--null` | Shorthand for --delimiter '\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out. [boolean] | +| | `--name-delimiter-tail` | Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name. [boolean] | +| | `--content-delimiter-tail` | Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe. [boolean] | +| | `--tail` | Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail. [boolean] | +| | `--nl` | Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed. [boolean] | +| `-H` | `--halt`, `--bail` | Halt on first error [boolean=false] | +| `-€` | `--void-euro` | Void having '€' as alias for '$'. The alias is used in all inputs, but never in globs, file names or piped data. [boolean] | +| `-§` | `--void-section` | Void having '§' as alias for '\'. The alias is used in all inputs, but never in globs, file names or piped data. [boolean] | +| `-o` | `--output` | Output the final result instead of saving to file. Will also output content even if no replacement has taken place. [boolean] | +| `-A` | `--void-async` | Handle files in a synchronous flow. Good to limit memory usage when handling large files. [boolean] | +| `-B` | `--void-backup` | Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted. [boolean] | +| `-b` | `--keep-backup` | Keep a backup file of the original content. With --rename the old file is kept. [boolean] | +| `-S` | `--simulate` | Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate. [boolean] | +| | `--sort` | Process files sync in a sorted order chosen from 6 values: `natural` (numbers compared as numbers and case insensitive), `ascii` (by char code, so `B` comes before `a`), `modified`, `created`, `size`, or `none` (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order. [natural, ascii, modified, created, size, none] =natural | +| | `--sort-reverse` | Turn the order of the files around. [boolean] | +| `-F` | `--expect-files` | Check that at least one file is found. The exit code is 1 if not. [boolean] | +| `-z` | `--expect-diff` | Check that at least one file is changed. The exit code is 1 if not. [boolean] | +| `-Z` | `--expect-zero-diff` | Check that no file is changed. The exit code is 1 if one is. [boolean] | +| | `--expect-diff-all` | Check that all files found are changed. The exit code is 1 if not. [boolean] | +| `-r` | `--rename` | Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots. [boolean] | +| | `--rename-path` | Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made. [boolean] | +| | `--rename-force` | With --rename or --rename-path, write over a file that is there already. [boolean] | +| `-m` | `--output-match` | Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. The replacement is not used, so --js code is not run. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). [boolean] | +| `-T` | `--trim-pipe` | Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean] | +| `-R` | `--replacement-pipe` | Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | +| | `--js-pipe` | Treat the replacement as javascript, and give the code the piped data as `pipe`. [boolean] | +| `-g` | `--glob-pipe` | Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] | +| | `--null-in` | The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out. [boolean] | +| `-j` | `--replacement-js`, `--js` | Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `file_path` and `file_path_rel`: the same as `file` and `file_rel`, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file, at its first match, unless it names the match, a group, `$m`, `$M` or `rr`. A file without a match does not run the code. Access a named group as `$` and the name: `(?...)` gives you `$year`. A group named like a number, like `n`, wins over the number. With the flag `n` of `-E regex+` only named groups capture, so the code has no `$1`, while a replacement text still reads `$1` as the first named group, as in javascript. `rr` has the values with long names: `filePath`, `filePathRel`, `dirPath`, `dirPathRel`, `dirName`, `fileName`, `name`, `ext`, `bytes`, `size`, `mtime`, `ctime`, `mtimeObj`, `ctimeObj`, `fileNumber`, `filesTotal`, `matchedFileNumber`, `changedFileNumber`, `matchThisFile`, `matchUntilNow`, `text`, `find`, `stdin`, `cwd`, `time` and `timeObj`, and the groups, like `rr[1]` and `rr.year`, which overwrite a name they share. Code that names `rr`, also in a string or a comment, runs once per match. `print()` and `printErr()`, also on `rr`, print to stdout and stderr, also with --quiet-total. `pad(value, width = 3, char = '0')` pads a value, like `pad($1)` for `007`. All variables, except from module, date objects, functions, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | +| `-c` | `--status-compact` | As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with `--quiet-total`. Will incude extra info about the inner workings of the engine in combination with the `--verbose` flag. [boolean] | +| | `--status` | Same as `--status-compact`, but always starting with a `\n` followed by pretty printed JSON. [boolean] | +| `-d` | `--debug` | Print only the status report, with what went into the engine and what it did. Does not block the output of --output and --output-match. [boolean] | +| `-h` | `--help` | Display help. [boolean] | ## Good to know @@ -589,8 +588,7 @@ The fuzzy tests get new input on every run, so they are not part of what the CI _**tl;dr**:_ _Files over 5 Mb are faster with `rr` than with `sed` - but - it does not matter as any file under 25 Mb has less than 0.7 seconds in difference._ -The speed test is initiated by `npm run test-speed`. The test takes files in different sizes and compares the processing time for RexReplace (`rr`) and the Unix tool `sed`. The test uses the sources of a website displaying [the book _1984_ by George Orwell](http://1984.surge.sh). The task for the tests is to remove all HTML tags by search-and-replace, so only the final text is left. The source is 888Kb, so all files up to 500Kb are generated directly from the source, while larger files are created by combining the first 500Kb several times. Each test runs 10 times to even out any temporary workload fluctuations. -Results from the latest speed test can always be seen in the [speed test log](https://github.com/mathiasrw/rexreplace/blob/main/test/speed/testlog.speed.md). +The speed test is `bash test/performance/tools/run.sh`, and [the README of the performance tests](test/performance/README.md) explains it and the others. It takes files in different sizes and compares the time RexReplace (`rr`) and other tools, like `sed`, take to remove all HTML tags from the HTML of [the book _1984_ by George Orwell](http://1984.surge.sh), so only the text is left.

diff --git a/npm/load-options.ts b/npm/load-options.ts new file mode 100644 index 00000000..4dc54194 --- /dev/null +++ b/npm/load-options.ts @@ -0,0 +1,28 @@ +// Writes the table of options in the README from the config of the CLI, so the README and the help say the same. +// Run by `yarn prerelease`. +import fs from 'fs'; +import path from 'path'; +import {argInfo} from '../../argmate/src/argMate.ts'; +import {argConfig} from '../src/options.ts'; + +const re = { + // From the first row under the headline Usage to the next headline, as a row can break over lines + optionsTable: /(\n## Usage\n[^]*?\n)\|[^]*?\n(?=\n## )/, +}; + +const file = path.join(import.meta.dir, '..', 'README.md'); +const help = argInfo({config: argConfig, format: 'markdown', showIntro: false, showOutro: false}); + +// Only the table, as the README has its own headline for it +const table = help.slice(help.indexOf('|')).trim() + '\n'; + +const readme = fs.readFileSync(file, 'utf8'); +if (!re.optionsTable.test(readme)) { + throw new Error('No table of options under the headline Usage in README.md'); +} + +// A function, as the table has `$1` and `$&` that a replacement string would read +fs.writeFileSync( + file, + readme.replace(re.optionsTable, (all, before) => before + table) +); diff --git a/package.json b/package.json index 01b0e36a..e7e5155e 100644 --- a/package.json +++ b/package.json @@ -49,9 +49,8 @@ "test-cli-bun": "AGENT=1 bash test/runtime/bun.sh", "test-cli-only": "AGENT=1 bash test/cli/run.sh && bash test/cli/run.ai.sh ", "test-ci": "yarn test-format && yarn build && yarn test-cli && yarn test-js && yarn test-minify", - "test-speed": "bash test/speed/run.sh", "rebase": "yarn is-git-clean && git fetch && git rebase origin/main", - "load-options": "# rr -h | rr 'Options:(.+)Examples:' _ -ms | rr '\\n {26,}|\\n\\n *' ' ' | rr \"'\" '`' | rr '^ (-.+?), (--[^ ]+) *' '`€1` | **`€2`** ' | rr '(^---- . ----).+?(## Good to know)' '€1 + nl + pipe + nl + nl + €2' readme.md -jspT", + "load-options": "bun npm/load-options.ts", "test-format": "biome format || (echo 'Please correct file formatting using `yarn format` and try again.' && exit 1)", "format": "biome format --write", "is-git-clean": "(git diff --quiet --exit-code --cached && git diff --quiet --exit-code) || (echo Please commit or stash changes && exit 1)", diff --git a/src/cli.ts b/src/cli.ts index ffc06282..d01bc7fc 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -5,20 +5,17 @@ import {writeTo} from './output.ts'; import {statusConfig, panic} from './status.ts'; import {alias} from './alias.ts'; import argMate, {argInfo} from '../../argmate/src/argMate.ts'; - -// Make it visible when Bun is the runtime (like with rexreplace-bun) as the regex engine is then not the one from Node. -// Added as semver build metadata (`+bun`) as a `-bun` would make it a prerelease. -const version = rexreplace.version + (process.versions.bun ? '+bun' : ''); +import {argConfig, version} from './options.ts'; let pattern, replacement; -// To avoid problems with patterns or replacements starting with '-' the two first arguments can not contain flags and are removed before yargs does it magic - but we still need to handle -version and -help +// To avoid problems with patterns or replacements starting with '-' the two first arguments can not contain flags and are removed before argMate does its magic - but we still need to handle --version and --help let needHelp = 0; if (process.argv.length < 4) { - if (/^(-v|--?version)$/.test(process.argv[process.argv.length - 1])) { + if (['-v', '--version'].includes(process.argv[process.argv.length - 1])) { console.log(version); process.exit(0); - } else if (/^(-h|--?help)$/.test(process.argv[process.argv.length - 1])) { + } else if (['-h', '--help'].includes(process.argv[process.argv.length - 1])) { needHelp = 1; } else { needHelp = 2; @@ -27,325 +24,6 @@ if (process.argv.length < 4) { [pattern, replacement] = process.argv.splice(2, 2); } -// The pattern and the replacement are taken out of argv above, so argMate only gets the rest -const argConfig = { - intro: [ - 'RexReplace ' + - version + - ': Regexp search and replace for files using lookahead and backreference to matching groups in the replacement. Defaults to global multiline case-insensitive search.', - ['Usage', '> rexreplace pattern replacement [fileGlob|option]+'], - [ - 'Examples', - ["> rexreplace 'Foo' 'xxx' myfile.md", "'foobar' in myfile.md will become 'xxxbar'"], - '', - ['> rr xxx Foo myfile.md', "The alias 'rr' can be used instead of 'rexreplace'"], - '', - [ - "> rexreplace '(f?(o))o(.*)' '$3$1€2' myfile.md", - "'foobar' in myfile.md will become 'barfoo'", - ], - '', - [ - "> rexreplace '^#' '##' *.md", - 'All markdown files in this dir got all headlines moved one level deeper', - ], - '', - [ - "> rexreplace 'a' 'b' 'myfile.md' 'src/**/*.*' ", - 'Provide multiple files or glob if needed', - ], - ], - ], - outro: "Inspiration: .oO(What should 'sed' have been by now?)", - version: { - type: 'boolean', - alias: ['v'], - description: 'Print rexreplace version (can be given as only argument)', - }, - verbose: {type: 'boolean', alias: ['V'], description: 'More chatty output'}, - literal: { - type: 'boolean', - alias: ['L'], - description: 'Literal string search (no regex used when searching)', - }, - literalReplacement: { - type: 'boolean', - alias: ['l'], - description: - 'No "smart" values in the replacement, so you can have text like `$1` and `$&` without escaping them.', - }, - replacementFile: { - type: 'boolean', - description: - 'The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path.', - }, - trimFile: {type: 'boolean', description: 'Trim the content of the replacement file.'}, - trim: {type: 'boolean', description: 'Shorthand for --trim-pipe and --trim-file.'}, - voidIgnoreCase: { - type: 'boolean', - alias: ['I'], - description: 'Void case insensitive search. Applies to the pattern and to all globs.', - }, - voidGlobal: { - type: 'boolean', - alias: ['G'], - description: 'Void global search (stop looking after the first match).', - }, - dotAll: {type: 'boolean', alias: ['s'], description: 'Have `.` also match newline.'}, - voidMultiline: { - type: 'boolean', - alias: ['M'], - description: - 'Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line.', - }, - unicode: { - type: 'boolean', - alias: ['u'], - description: - 'Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode.', - }, - voidUnicode: { - type: 'boolean', - alias: ['U'], - description: - 'Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode.', - }, - flags: { - type: 'string', - description: - 'The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U.', - conflict: ['I', 'G', 'M', 's', 'u', 'U'], - }, - encoding: { - default: 'utf8', - alias: ['e'], - description: 'Encoding of files/piped data.', - type: 'string', - validate: value => Buffer.isEncoding(value) || `Unknown encoding: ${value}`, - }, - excludeRe: { - type: 'string[]', - alias: ['x'], - description: - 'Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times.', - }, - excludeGlob: { - type: 'string[]', - alias: ['X'], - description: - 'Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate.', - }, - engine: { - valid: [['rr', 'default'], 'regex+', 'runtime'], - alias: ['E'], - description: - 'The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun.', - }, - quiet: { - type: 'boolean', - alias: ['q'], - description: - 'Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown.', - }, - quietTotal: {type: 'boolean', alias: ['Q'], description: 'Never display errors or info'}, - fullPath: { - type: 'boolean', - alias: ['f'], - description: 'Print the full path of the named files instead of the relative path.', - }, - nameAll: { - type: 'boolean', - description: - 'Print the names of all files found by the globs, also the ones where nothing was changed.', - }, - nameDelimiter: { - type: 'string', - description: - 'String printed between the names of the files. Default is a line break, unless the name already ends with one.', - }, - contentDelimiter: { - type: 'string', - description: - 'String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one.', - }, - delimiter: { - type: 'string', - description: - 'Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\\t`, `\\n` and the `€` and `§` aliases.', - }, - nullOut: { - type: 'boolean', - alias: ['null'], - description: - "Shorthand for --delimiter '\\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out.", - }, - nameDelimiterTail: { - type: 'boolean', - description: - 'Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name.', - }, - contentDelimiterTail: { - type: 'boolean', - description: - 'Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe.', - }, - tail: { - type: 'boolean', - description: - 'Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail.', - }, - nl: { - type: 'boolean', - description: - 'Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed.', - }, - halt: {type: 'boolean', default: false, alias: ['H', 'bail'], description: 'Halt on first error'}, - voidEuro: { - type: 'boolean', - alias: ['€'], - description: - "Void having '€' as alias for '$'. The alias is used in all inputs, but never in globs, file names or piped data.", - }, - voidSection: { - type: 'boolean', - alias: ['§'], - description: - "Void having '§' as alias for '\\'. The alias is used in all inputs, but never in globs, file names or piped data.", - }, - output: { - type: 'boolean', - alias: ['o'], - description: - 'Output the final result instead of saving to file. Will also output content even if no replacement has taken place.', - }, - voidAsync: { - type: 'boolean', - alias: ['A'], - description: - 'Handle files in a synchronous flow. Good to limit memory usage when handling large files. ', - }, - voidBackup: { - type: 'boolean', - alias: ['B'], - description: - 'Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted.', - }, - keepBackup: { - type: 'boolean', - alias: ['b'], - description: 'Keep a backup file of the original content. With --rename the old file is kept.', - }, - simulate: { - type: 'boolean', - alias: ['S'], - description: - 'Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate.', - }, - sort: { - type: 'string', - valid: ['natural', 'ascii', 'modified', 'created', 'size', 'none'], - description: - 'Process files sync in a sorted order chosen from 6 values: `natural` (numbers compared as numbers and case insensitive), `ascii` (by char code, so `B` comes before `a`), `modified`, `created`, `size`, or `none` (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order.', - transform: value => String(value).toLowerCase(), - }, - sortReverse: {type: 'boolean', description: 'Turn the order of the files around.'}, - expectFiles: { - type: 'boolean', - alias: ['F'], - description: 'Check that at least one file is found. The exit code is 1 if not.', - }, - expectDiff: { - type: 'boolean', - alias: ['z'], - description: 'Check that at least one file is changed. The exit code is 1 if not.', - }, - expectZeroDiff: { - type: 'boolean', - alias: ['Z'], - description: 'Check that no file is changed. The exit code is 1 if one is.', - conflict: ['z', 'expect-diff-all'], - }, - expectDiffAll: { - type: 'boolean', - description: 'Check that all files found are changed. The exit code is 1 if not.', - }, - rename: { - type: 'boolean', - alias: ['r'], - description: - 'Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots.', - conflict: ['rename-path'], - }, - renamePath: { - type: 'boolean', - description: - 'Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made.', - }, - renameForce: { - type: 'boolean', - description: 'With --rename or --rename-path, write over a file that is there already.', - }, - outputMatch: { - type: 'boolean', - alias: ['m'], - description: - 'Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. The replacement is not used, so --js code is not run. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). ', - }, - trimPipe: { - type: 'boolean', - alias: ['T'], - description: - 'Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. ', - }, - replacementPipe: { - type: 'boolean', - alias: ['R'], - description: - 'Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter.', - }, - jsPipe: { - type: 'boolean', - description: 'Treat the replacement as javascript, and give the code the piped data as `pipe`.', - }, - globPipe: { - type: 'boolean', - alias: ['g'], - description: - 'Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments.', - conflict: ['R', 'js-pipe'], - }, - nullIn: { - type: 'boolean', - description: - 'The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out.', - demand: ['glob-pipe'], - }, - replacementJs: { - type: 'boolean', - alias: ['j', 'js'], - description: - "Treat replacement as javascript source code. \nThe statement from the last expression will become the replacement string. \nPurposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. \nThe full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. \nAt some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. \n\nThe code has access to the following variables: \n`r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, \n`fs` from node, \n`path` from node, \n`globs` from npm, \n`pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), \n`find`: pattern searched for (the needle), \n`text`: full text being searched i.e. file content or piped data (the haystack), \n`bytes`: total size of the haystack in bytes, \n`size`: human-friendly representation of the total size of the haystack, \n`time`: String representing the local time when the command was invoked,\n`time_obj`: date object representing `time`,\n`now`: alias for `time`,\n`cwd`: current process working dir, \n`nl`: a new-line char,\n`_`: a single space char (for easy string concatenation).\n\nThe following values defaults to `❌` if haystack does not originate from a file:\n`file`: contains the full path of the active file being searched (including full filename), \n`file_rel`: contains `file` relative to current process working dir, \n`file_path` and `file_path_rel`: the same as `file` and `file_rel`, \n`dirpath`: contains the full path without filename of the active file being searched, \n`dirpath_rel`: contains `dirpath` relative to current process working dir, \n`filename`: is the full filename of the active file being searched without path, \n`name`: filename of the active file being searched with no extension, \n`ext`: extension of the filename including leading dot, \n`mtime`: ISO inspired representation of the last local modification time of the current file, \n`ctime`: ISO representation of the local creation time of the current file. \n`mtime_obj`: date object representing `mtime`, \n`ctime_obj`: date object representing `ctime`.\n\nThe numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file, at its first match, unless it names the match, a group, `$m`, `$M` or `rr`. A file without a match does not run the code.\n\nAccess a named group as `$` and the name: `(?...)` gives you `$year`. A group named like a number, like `n`, wins over the number. With the flag `n` of `-E regex+` only named groups capture, so the code has no `$1`, while a replacement text still reads `$1` as the first named group, as in javascript. \n\n`rr` has the values with long names: `filePath`, `filePathRel`, `dirPath`, `dirPathRel`, `dirName`, `fileName`, `name`, `ext`, `bytes`, `size`, `mtime`, `ctime`, `mtimeObj`, `ctimeObj`, `fileNumber`, `filesTotal`, `matchedFileNumber`, `changedFileNumber`, `matchThisFile`, `matchUntilNow`, `text`, `find`, `stdin`, `cwd`, `time` and `timeObj`, and the groups, like `rr[1]` and `rr.year`, which overwrite a name they share. Code that names `rr`, also in a string or a comment, runs once per match. `print()` and `printErr()`, also on `rr`, print to stdout and stderr, also with --quiet-total. `pad(value, width = 3, char = '0')` pads a value, like `pad($1)` for `007`.\n\nAll variables, except from module, date objects, functions, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). \n", - }, - statusCompact: { - type: 'boolean', - alias: ['c'], - description: - 'As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with `--quiet-total`. Will incude extra info about the inner workings of the engine in combination with the `--verbose` flag.', - }, - status: { - type: 'boolean', - description: - 'Same as `--status-compact`, but always starting with a `\\n` followed by pretty printed JSON.', - }, - debug: { - type: 'boolean', - alias: ['d'], - description: - 'Print only the status report, with what went into the engine and what it did. Does not block the output of --output and --output-match.', - }, - help: {type: 'boolean', alias: ['h'], description: 'Display help.'}, -}; - // The engine resolves with the status of the run, or throws a panic when the run can not go on. // The CLI is the only part that acts on the process. function printStatus(config, status) { @@ -369,7 +47,7 @@ function exitOnPanic(config, e) { // Help that was asked for goes to stdout, and the help with an error to stderr function backOut(exitcode = 1) { - (0 === exitcode ? console.log : console.error)(argInfo({}, argConfig)); + (0 === exitcode ? console.log : console.error)(argInfo({config: argConfig})); process.exit(exitcode); } @@ -396,6 +74,7 @@ if (0 < needHelp) { backOut(1 === needHelp ? 0 : 3); } +// The pattern and the replacement are taken out of argv above, so argMate only gets the rest. // Wrong flags are a panic, so the exit code is 3. The help and the message go to stderr. const argv = argMate(process.argv.slice(2), argConfig, { allowUnknown: false, diff --git a/src/options.ts b/src/options.ts new file mode 100644 index 00000000..efc5fdbb --- /dev/null +++ b/src/options.ts @@ -0,0 +1,422 @@ +// The options of the CLI, for its parse and its help, and for the table of options in the README + +import {version as engineVersion} from './engine.ts'; + +// Make it visible when Bun is the runtime (like with rexreplace-bun) as the regex engine is then not the one from Node. +// Added as semver build metadata (`+bun`) as a `-bun` would make it a prerelease. +export const version = engineVersion + (process.versions.bun ? '+bun' : ''); + +export const argConfig = { + intro: [ + 'RexReplace ' + + version + + ': Regexp search and replace for files using lookahead and backreference to matching groups in the replacement. Defaults to global multiline case-insensitive search.', + ['Usage', '> rexreplace pattern replacement [fileGlob|option]+'], + [ + 'Examples', + ["> rexreplace 'Foo' 'xxx' myfile.md", "'foobar' in myfile.md will become 'xxxbar'"], + '', + ['> rr xxx Foo myfile.md', "The alias 'rr' can be used instead of 'rexreplace'"], + '', + [ + "> rexreplace '(f?(o))o(.*)' '$3$1€2' myfile.md", + "'foobar' in myfile.md will become 'barfoo'", + ], + '', + [ + "> rexreplace '^#' '##' *.md", + 'All markdown files in this dir got all headlines moved one level deeper', + ], + '', + [ + "> rexreplace 'a' 'b' 'myfile.md' 'src/**/*.*' ", + 'Provide multiple files or glob if needed', + ], + ], + ], + outro: "Inspiration: .oO(What should 'sed' have been by now?)", + version: { + type: 'boolean', + alias: ['v'], + description: 'Print rexreplace version (can be given as only argument)', + }, + verbose: { + type: 'boolean', + alias: ['V'], + description: 'More chatty output', + // conflict: ['q', 'Q'], + }, + literal: { + type: 'boolean', + alias: ['L'], + description: 'Literal string search (no regex used when searching)', + }, + literalReplacement: { + type: 'boolean', + alias: ['l'], + description: + 'No "smart" values in the replacement, so you can have text like `$1` and `$&` without escaping them.', + }, + replacementFile: { + type: 'boolean', + description: + 'The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path.', + }, + trimFile: {type: 'boolean', description: 'Trim the content of the replacement file.'}, + trim: {type: 'boolean', description: 'Shorthand for --trim-pipe and --trim-file.'}, + voidIgnoreCase: { + type: 'boolean', + alias: ['I'], + description: 'Void case insensitive search. Applies to the pattern and to all globs.', + }, + voidGlobal: { + type: 'boolean', + alias: ['G'], + description: 'Void global search (stop looking after the first match).', + }, + dotAll: {type: 'boolean', alias: ['s'], description: 'Have `.` also match newline.'}, + voidMultiline: { + type: 'boolean', + alias: ['M'], + description: + 'Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line.', + }, + unicode: { + type: 'boolean', + alias: ['u'], + description: + 'Treat patterns as a sequence of unicode code points. For the default engine it is on by default. For the regex+ engine (`-E regex+`) it swaps from v mode to u mode.', + }, + voidUnicode: { + type: 'boolean', + alias: ['U'], + description: + 'Void unicode. For the default engine it means reading the pattern as UTF-16 parts (escapes of chars that need no escaping are allowed). For the regex+ engine it has no impact, as it defaults to v mode.', + }, + flags: { + type: 'string', + description: + 'The flags of the regex, like `gimsv`, in place of the defaults: `gimu`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s, -u or -U.', + conflict: ['I', 'G', 'M', 's', 'u', 'U'], + }, + // Not `valid` as node takes the name of an encoding in any case, like `UTF8` + encoding: { + default: 'utf8', + alias: ['e'], + description: 'Encoding of files/piped data.', + type: 'string', + validate: value => Buffer.isEncoding(value) || `Unknown encoding: ${value}`, + }, + excludeRe: { + type: 'string[]', + alias: ['x'], + description: + 'Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times.', + }, + excludeGlob: { + type: 'string[]', + alias: ['X'], + description: + 'Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate.', + }, + engine: { + valid: [['rr', 'default'], 'regex+', 'runtime'], + alias: ['E'], + description: + 'The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun.', + }, + quiet: { + type: 'boolean', + alias: ['q'], + description: + 'Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown.', + }, + quietTotal: {type: 'boolean', alias: ['Q'], description: 'Never display errors or info'}, + fullPath: { + type: 'boolean', + alias: ['f'], + description: 'Print the full path of the named files instead of the relative path.', + }, + nameAll: { + type: 'boolean', + description: + 'Print the names of all files found by the globs, also the ones where nothing was changed.', + }, + nameDelimiter: { + type: 'string', + description: + 'String printed between the names of the files. Default is a line break, unless the name already ends with one.', + }, + contentDelimiter: { + type: 'string', + description: + 'String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one.', + }, + delimiter: { + type: 'string', + description: + 'Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\\t`, `\\n` and the `€` and `§` aliases.', + }, + nullOut: { + type: 'boolean', + alias: ['null'], + description: + "Shorthand for --delimiter '\\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out.", + }, + nameDelimiterTail: { + type: 'boolean', + description: + 'Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name.', + }, + contentDelimiterTail: { + type: 'boolean', + description: + 'Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe.', + }, + tail: { + type: 'boolean', + description: + 'Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail.', + }, + nl: { + type: 'boolean', + description: + 'Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed.', + }, + halt: {type: 'boolean', default: false, alias: ['H', 'bail'], description: 'Halt on first error'}, + voidEuro: { + type: 'boolean', + alias: ['€'], + description: + "Void having '€' as alias for '$'. The alias is used in all inputs, but never in globs, file names or piped data.", + }, + voidSection: { + type: 'boolean', + alias: ['§'], + description: + "Void having '§' as alias for '\\'. The alias is used in all inputs, but never in globs, file names or piped data.", + }, + output: { + type: 'boolean', + alias: ['o'], + description: + 'Output the final result instead of saving to file. Will also output content even if no replacement has taken place.', + // conflict: ['O'], + }, + voidAsync: { + type: 'boolean', + alias: ['A'], + description: + 'Handle files in a synchronous flow. Good to limit memory usage when handling large files. ', + }, + voidBackup: { + type: 'boolean', + alias: ['B'], + description: + 'Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted.', + }, + keepBackup: { + type: 'boolean', + alias: ['b'], + description: 'Keep a backup file of the original content. With --rename the old file is kept.', + }, + simulate: { + type: 'boolean', + alias: ['S'], + description: + 'Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate.', + }, + sort: { + type: 'string', + valid: ['natural', 'ascii', 'modified', 'created', 'size', 'none'], + description: + 'Process files sync in a sorted order chosen from 6 values: `natural` (numbers compared as numbers and case insensitive), `ascii` (by char code, so `B` comes before `a`), `modified`, `created`, `size`, or `none` (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order.', + // Any case, and true and yes are natural + transform: value => { + const name = String(value).toLowerCase(); + return 'true' === name || 'yes' === name ? 'natural' : name; + }, + }, + sortReverse: {type: 'boolean', description: 'Turn the order of the files around.'}, + // Checks for a CI step. When one is not met, the exit code is 1. + expectFiles: { + type: 'boolean', + alias: ['F'], + description: 'Check that at least one file is found. The exit code is 1 if not.', + }, + expectDiff: { + type: 'boolean', + alias: ['z'], + description: 'Check that at least one file is changed. The exit code is 1 if not.', + }, + expectZeroDiff: { + type: 'boolean', + alias: ['Z'], + description: 'Check that no file is changed. The exit code is 1 if one is.', + conflict: ['z', 'expect-diff-all'], + }, + expectDiffAll: { + type: 'boolean', + description: 'Check that all files found are changed. The exit code is 1 if not.', + }, + rename: { + type: 'boolean', + alias: ['r'], + description: + 'Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots.', + conflict: ['rename-path'], + }, + renamePath: { + type: 'boolean', + description: + 'Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made.', + }, + renameForce: { + type: 'boolean', + description: 'With --rename or --rename-path, write over a file that is there already.', + }, + outputMatch: { + type: 'boolean', + alias: ['m'], + description: + 'Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. The replacement is not used, so --js code is not run. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). ', + }, + trimPipe: { + type: 'boolean', + alias: ['T'], + description: + 'Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. ', + }, + replacementPipe: { + type: 'boolean', + alias: ['R'], + description: + 'Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter.', + }, + jsPipe: { + type: 'boolean', + description: 'Treat the replacement as javascript, and give the code the piped data as `pipe`.', + }, + globPipe: { + type: 'boolean', + alias: ['g'], + description: + 'Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments.', + // The piped data can only be one thing + conflict: ['R', 'js-pipe'], + }, + nullIn: { + type: 'boolean', + description: + 'The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out.', + demand: ['glob-pipe'], + }, + replacementJs: { + type: 'boolean', + alias: ['j', 'js'], + description: + "Treat replacement as javascript source code. \nThe statement from the last expression will become the replacement string. \nPurposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. \nThe full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. \nAt some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. \n\nThe code has access to the following variables: \n`r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, \n`fs` from node, \n`path` from node, \n`globs` from npm, \n`pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), \n`find`: pattern searched for (the needle), \n`text`: full text being searched i.e. file content or piped data (the haystack), \n`bytes`: total size of the haystack in bytes, \n`size`: human-friendly representation of the total size of the haystack, \n`time`: String representing the local time when the command was invoked,\n`time_obj`: date object representing `time`,\n`now`: alias for `time`,\n`cwd`: current process working dir, \n`nl`: a new-line char,\n`_`: a single space char (for easy string concatenation).\n\nThe following values defaults to `❌` if haystack does not originate from a file:\n`file`: contains the full path of the active file being searched (including full filename), \n`file_rel`: contains `file` relative to current process working dir, \n`file_path` and `file_path_rel`: the same as `file` and `file_rel`, \n`dirpath`: contains the full path without filename of the active file being searched, \n`dirpath_rel`: contains `dirpath` relative to current process working dir, \n`filename`: is the full filename of the active file being searched without path, \n`name`: filename of the active file being searched with no extension, \n`ext`: extension of the filename including leading dot, \n`mtime`: ISO inspired representation of the last local modification time of the current file, \n`ctime`: ISO representation of the local creation time of the current file. \n`mtime_obj`: date object representing `mtime`, \n`ctime_obj`: date object representing `ctime`.\n\nThe numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file, at its first match, unless it names the match, a group, `$m`, `$M` or `rr`. A file without a match does not run the code.\n\nAccess a named group as `$` and the name: `(?...)` gives you `$year`. A group named like a number, like `n`, wins over the number. With the flag `n` of `-E regex+` only named groups capture, so the code has no `$1`, while a replacement text still reads `$1` as the first named group, as in javascript. \n\n`rr` has the values with long names: `filePath`, `filePathRel`, `dirPath`, `dirPathRel`, `dirName`, `fileName`, `name`, `ext`, `bytes`, `size`, `mtime`, `ctime`, `mtimeObj`, `ctimeObj`, `fileNumber`, `filesTotal`, `matchedFileNumber`, `changedFileNumber`, `matchThisFile`, `matchUntilNow`, `text`, `find`, `stdin`, `cwd`, `time` and `timeObj`, and the groups, like `rr[1]` and `rr.year`, which overwrite a name they share. Code that names `rr`, also in a string or a comment, runs once per match. `print()` and `printErr()`, also on `rr`, print to stdout and stderr, also with --quiet-total. `pad(value, width = 3, char = '0')` pads a value, like `pad($1)` for `007`.\n\nAll variables, except from module, date objects, functions, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). \n", + }, + statusCompact: { + type: 'boolean', + alias: ['c'], + description: + 'As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with `--quiet-total`. Will incude extra info about the inner workings of the engine in combination with the `--verbose` flag.', + }, + status: { + type: 'boolean', + description: + 'Same as `--status-compact`, but always starting with a `\\n` followed by pretty printed JSON.', + }, + debug: { + type: 'boolean', + alias: ['d'], + description: + 'Print only the status report, with what went into the engine and what it did. Does not block the output of --output and --output-match.', + }, + help: {type: 'boolean', alias: ['h'], description: 'Display help.'}, + + /* + voidNewline: { + type: 'boolean', + alias: ['N'], + description: + `Avoid having newline when outputting data (or when piping). `+ + `Normally . `+ + '' + }, + */ + + /* + patternFile: { + type: 'boolean', + alias: ['P'], + description: "Pattern is a filename from where the pattern will be generated. If more than one line is found in the file the pattern will be defined by each line trimmed and having newlines removed followed by other all rules (like -€).)", + }, + + replacementFile: { + type: 'boolean', + alias: ['R'], + description: + `Replacement is a filename from where the replacement will be generated. ` + + `If more than one line is found in the file the final replacement will be defined by each line trimmed and having newlines removed followed by all other rules (like -€).` + }, + */ + + /* // Ideas + + // https://github.com/eugeneware/replacestream + maxMatchLen: { + type: 'int', + alias: ['M'], + description: "Maximum length of match. Set this value only if any single file of your ", + default: false, + }, + + + + globsFile: { + type: 'boolean', + alias: ['G'], + description: "filename/globas are filename(s) for files containing one filename/globs on each line to be search/replaced", + }, + + patternJs: { + type: 'boolean', + alias: ['j'], + description: "Pattern is javascript source that will return a string giving the pattern to use", + }, + + + globJs: { + type: 'boolean', + description: "filename/globs are javascript source that will return a string with newline seperating each glob to work on", + }, + + + + -S --stream-size + + -l --stream-line + + + + + +Upper Case Free Lower Case Free + a +C +D + i +J +K k +N n +O +P p + t +W w +Y y + + + */ +}; diff --git a/test/cli/run.sh b/test/cli/run.sh index ecebb957..97e38ee3 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -280,6 +280,13 @@ echo 'foo' > aab.file echo 'foo' > b.file assert "LC_ALL=da_DK.UTF-8 rexreplace 'foo' 'x' b.file aab.file --sort natural --simulate --name-delimiter ','" 'aab.file,b.file,' +# true and yes in any case are natural +reset +echo 'foo' > a10.file +echo 'foo' > a9.file +assert "rexreplace 'foo' 'x' a10.file a9.file --sort YES --simulate --name-delimiter ','" 'a9.file,a10.file,' +assert "rexreplace 'foo' 'x' a10.file a9.file --sort true --simulate --name-delimiter ','" 'a9.file,a10.file,' + # --full-path prints the full path of each name reset diff --git a/test/fix/692.test.js b/test/fix/692.test.js index decbc4fc..87257125 100644 --- a/test/fix/692.test.js +++ b/test/fix/692.test.js @@ -5,21 +5,33 @@ import {$} from 'bun'; import {cli, sh} from '../helper.js'; describe('#692 a single argument', () => { - test.each(['-v', '-version', '--version'])('%s prints only the version', async arg => { + test.each(['-v', '--version'])('%s prints only the version', async arg => { const result = await sh($`bun ${cli} ${arg}`); expect(result).toEqual({out: (await sh($`bun ${cli} --version`)).out, err: '', exit: 0}); expect(result.out).toMatch(/^\S+\n$/); }); - test.each(['-h', '-help', '--help'])('%s prints the help to stdout', async arg => { + test.each(['-h', '--help'])('%s prints the help to stdout', async arg => { const result = await sh($`bun ${cli} ${arg}`); expect(result.out).toContain('Options:'); expect(result.err).toBe(''); expect(result.exit).toBe(0); }); - // -V is verbose and -H is halt: other flags, not a request for the version or the help - test.each(['my-vile', '-V', 'o-ho', '-H', 'a-v', 'a-h', 'xversion', 'xhelp'])( + // -V is verbose and -H is halt, other flags and not a request for the version or the help. + // -version and -help are the short flags -v -e -r -s -i -o -n and -h -e -l -p, as for argmate. + test.each([ + 'my-vile', + '-V', + 'o-ho', + '-H', + 'a-v', + 'a-h', + 'xversion', + 'xhelp', + '-version', + '-help', + ])( '%s is wrong use: the help goes to stderr, nothing to stdout, and the exit code is 3', async arg => { const result = await sh($`bun ${cli} ${arg}`); From 6d36dc697b6069d1dcc443f51f11a35057357f85 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Sun, 1 Nov 2026 00:05:04 +1100 Subject: [PATCH 6/9] Check each tool before timing it, compare all majors of rr in readme-speed, and give each row its own 1.00x --- test/performance/README.md | 23 +++++++- test/performance/readme-speed/run.sh | 73 ++++++++++++++++++++++++ test/performance/shared/build.sh | 10 ++++ test/performance/shared/check.sh | 18 ++++++ test/performance/shared/table.js | 33 +++++++++++ test/performance/tools/run.sh | 47 ++++++++++----- test/performance/yargs-vs-argmate/run.sh | 34 ++++++----- 7 files changed, 203 insertions(+), 35 deletions(-) create mode 100755 test/performance/readme-speed/run.sh create mode 100644 test/performance/shared/build.sh create mode 100644 test/performance/shared/check.sh create mode 100644 test/performance/shared/table.js diff --git a/test/performance/README.md b/test/performance/README.md index 791cf48a..3a322527 100644 --- a/test/performance/README.md +++ b/test/performance/README.md @@ -2,23 +2,40 @@ These tests measure how fast RexReplace is. They are not part of `yarn test`, because the times depend on the machine and on what else it is doing. Run them by hand, and compare numbers from the same machine only. -Each test has its own folder with a `run.sh`. It builds rr from the working tree into a temp folder, runs [hyperfine](https://github.com/sharkdp/hyperfine) and prints the results. The runs start from the repo root, a usual place to run rr. +Each test has its own folder with a `run.sh`. It builds rr from the working tree into a temp folder, runs [hyperfine](https://github.com/sharkdp/hyperfine), and ends with two tables. The first gives each time relative to the fastest in its row, and the second the seconds for one run. Both use the median, as one slow run from other work on the machine moves the mean. The runs start from the repo root, a usual place to run rr. + +`shared` has what the tests have in common, the build of rr (`build.sh`), the check that each tool gives the same file (`check.sh`) and the tables (`table.js`). + +Before the timing of each size, every command runs once, and the test stops if one fails or gives another file than the others. A tool that does other work, or no work, would else be timed for it. The file to change is in the ignored `tmp` folder of the repo and is given by a path from the repo, as rr 3 to 7 can not write to an absolute path outside the working folder (#726). They say nothing and end with exit code 0 (#728), so without the check they look fast on big files. ## tools -Compares rr with other tools that replace in files. Each tool removes the HTML tags of the book 1984 by George Orwell (`1984.html`) in place, on files of 1 KB, 10 KB, 100 KB, 1 MB, 10 MB and 100 MB. +Compares rr with the other tools that replace in files, from the list of related projects in the README. Each tool removes the HTML tags of the book 1984 by George Orwell (`1984.html`) in place, on files from 1 KB to 100 MB. ```bash bash test/performance/tools/run.sh bash test/performance/tools/run.sh 1KB 10MB ``` -The tools are rr on Node, rr on Bun, `sed`, `gsed`, `perl`, `sd` and `replace-in-file`. A tool that is not installed is left out. +The tools are rr on Node, rr on Bun, `sed`, `gsed`, `perl`, `sd`, `replace-in-file`, `replace-x` and `replace`. The ones from npm are installed in the temp folder for the run. `gsed` and `sd` are left out when they are not installed (`brew install gnu-sed sd`). `sed` and `perl` read a line at a time, so a tag that goes over two lines is not removed. rr and the other tools read the whole file, so their pattern stops at a line break. That way all the tools do the same work, and the files end up the same. A small file shows the start of each tool, and a big file shows the regex engine and the reading and writing of the file. +## readme-speed + +Runs the speed test of the README again, so its numbers can be checked. rr runs against `sed` on the same sizes as the table there, from 1 KB to 100 MB, with the last release of each major of rr from npm, and with rr from the working tree on Node and on Bun. + +```bash +bash test/performance/readme-speed/run.sh +bash test/performance/readme-speed/run.sh 1KB 10MB +``` + +The files are made as the README says. Up to 500 KB they are cut from the book, and a larger file is the first 500 KB put together again and again. Each command runs 10 times, as in the README. The pattern stops at a line break for rr, as in the tools test. + +The releases are installed in the temp folder for the run. rr 6 needs `re2`, and the version it asks for does not build on a new Node, so a later `re2` stands in for it. + ## yargs-vs-argmate Compares the start of the CLI with yargs against the CLI of the working tree with argmate, on Node and on Bun. Most of a short run is the start, so this is where the parser of the flags shows. diff --git a/test/performance/readme-speed/run.sh b/test/performance/readme-speed/run.sh new file mode 100755 index 00000000..8de551c0 --- /dev/null +++ b/test/performance/readme-speed/run.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash + +# Runs the speed test of the README again, rr against sed on the sizes of its table, with the last release of each major of rr. +# The files are made as the README says. Up to 500 KB they are cut from the book, and a larger file is the first 500 KB put together again and again. +# Give sizes as arguments to run only those, like `1KB 10MB`. + +set -e + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$DIR/../../.." && pwd)" +SIZES="${*:-1KB 5KB 10KB 100KB 500KB 1MB 5MB 10MB 25MB 50MB 100MB}" +VERSIONS='1.1.0 2.5.3 3.1.1 4.1.2 5.3.0 6.1.5 7.1.15' + +# The file to change is in tmp of the repo, and given by a path from the repo, as rr 3 to 7 can not write to an absolute path outside the working folder +mkdir -p "$ROOT/tmp" +TMP="$(mktemp -d)" +WORK="$(mktemp -d "$ROOT/tmp/performance.XXXXXX")" +trap 'rm -rf "$TMP" "$WORK"' EXIT + +source "$DIR/../shared/build.sh" +source "$DIR/../shared/check.sh" +build_rr "$ROOT" "$TMP/rr" + +# rr 6 needs re2, and the version it asks for does not build on a new Node, so a later re2 stands in for it +for version in $VERSIONS; do + mkdir "$TMP/$version" + echo "{\"dependencies\": {\"rexreplace\": \"$version\"}, \"overrides\": {\"re2\": \"1.22.1\"}}" > "$TMP/$version/package.json" + (cd "$TMP/$version" && npm install --silent --no-audit --no-fund > /dev/null) +done + +# sed reads a line at a time, so a tag over two lines stays. rr reads the whole file, so its pattern stops at a line break to do the same work. +TAGS='<[^>]*>' +TAGS_IN_A_LINE='<[^>\n]*>' +FILE="${WORK#"$ROOT/"}/test.file" + +# BSD sed (macOS) needs an empty suffix after -i, GNU sed does not take one +if sed --version > /dev/null 2>&1; then + SED_INPLACE="-i" +else + SED_INPLACE="-i ''" +fi + +ARGS=(-n 'sed' "sed $SED_INPLACE 's/$TAGS//g' $FILE") +for version in $VERSIONS; do + ARGS+=(-n "rr $version" "node $TMP/$version/node_modules/.bin/rexreplace '$TAGS_IN_A_LINE' '' $FILE") +done +ARGS+=(-n 'rr now' "node $TMP/rr/node.mjs '$TAGS_IN_A_LINE' '' $FILE") +ARGS+=(-n 'rr now on Bun' "bun $TMP/rr/bun.mjs '$TAGS_IN_A_LINE' '' $FILE") + +head -c $((500 * 1024)) "$DIR/../tools/1984.html" > "$TMP/500KB.part" + +# The runs start from the repo, a usual place for rr +cd "$ROOT" + +for size in $SIZES; do + number="${size//[!0-9]/}" + unit="${size//[0-9]/}" + case "$unit" in + KB) head -c $((number * 1024)) "$DIR/../tools/1984.html" > "$TMP/$size.file" ;; + MB) for i in $(seq $((number * 2))); do cat "$TMP/500KB.part"; done > "$TMP/$size.file" ;; + *) echo "Unknown size $size, use KB or MB like 10KB" && exit 1 ;; + esac + + cp "$TMP/$size.file" "$TMP/$size.reference" + perl -pi -e "s/$TAGS//g" "$TMP/$size.reference" + check_same "$TMP/$size.file" "$FILE" "$TMP/$size.reference" "${ARGS[@]}" + + echo '' + echo "## $size" + hyperfine -N --warmup 1 --runs 10 --prepare "cp $TMP/$size.file $FILE" --export-json "$TMP/$size.json" "${ARGS[@]}" +done + +node "$DIR/../shared/table.js" $(for size in $SIZES; do echo "$TMP/$size.json"; done) diff --git a/test/performance/shared/build.sh b/test/performance/shared/build.sh new file mode 100644 index 00000000..e2125fe6 --- /dev/null +++ b/test/performance/shared/build.sh @@ -0,0 +1,10 @@ +# Builds the CLI of a source tree for Node and for Bun, as node.mjs and bun.mjs in a folder. +# The packages are external, so the folder gets a node_modules from the repo. +# .mjs as the folder has no package.json that says the files are ES modules. +# build_rr +build_rr() { + mkdir -p "$2" + ln -sfn "$ROOT/node_modules" "$2/node_modules" + (cd "$1" && bun build src/cli.node.ts --target=node --format=esm --packages=external --outfile="$2/node.mjs" > /dev/null) + (cd "$1" && bun build src/cli.bun.ts --target=bun --format=esm --packages=external --outfile="$2/bun.mjs" > /dev/null) +} diff --git a/test/performance/shared/check.sh b/test/performance/shared/check.sh new file mode 100644 index 00000000..f144df37 --- /dev/null +++ b/test/performance/shared/check.sh @@ -0,0 +1,18 @@ +# Runs each command once on a copy of a file, and stops the test when one fails or does not give the same file as the reference. +# Else a tool that does other work, or no work, would be timed for it. +# check_same -n [-n ]... +check_same() { + local source="$1" file="$2" reference="$3" + shift 3 + while [ $# -gt 0 ]; do + local name="$2" command="$3" + shift 3 + cp "$source" "$file" + if ! eval "$command" > /dev/null 2>&1 < /dev/null; then + echo "$name failed" && exit 1 + fi + if ! cmp -s "$file" "$reference"; then + echo "$name does not give the same file as the others" && exit 1 + fi + done +} diff --git a/test/performance/shared/table.js b/test/performance/shared/table.js new file mode 100644 index 00000000..9d98b557 --- /dev/null +++ b/test/performance/shared/table.js @@ -0,0 +1,33 @@ +// Prints the results of hyperfine as two markdown tables, one row for each JSON file of `hyperfine --export-json`. +// The name of the file is the name of the row, and the names of the commands are the columns. +// The median, as one slow run from other work on the machine moves the mean. +// node table.js ... +import {readFileSync} from 'node:fs'; +import {basename} from 'node:path'; + +const rows = process.argv.slice(2).map(file => ({ + name: basename(file, '.json'), + results: JSON.parse(readFileSync(file, 'utf8')).results, +})); +const columns = [...new Set(rows.flatMap(row => row.results.map(result => result.command)))]; + +function table(title, cell) { + console.log(`\n${title}\n`); + console.log(`| | ${columns.join(' | ')} |`); + console.log(`|---|${columns.map(() => '---:').join('|')}|`); + for (const {name, results} of rows) { + const fastest = Math.min(...results.map(result => result.median)); + const cells = columns.map(column => { + const result = results.find(result => result.command === column); + return result ? cell(result.median, fastest) : ''; + }); + console.log(`| ${name} | ${cells.join(' | ')} |`); + } +} + +// The same number of decimals in each table, so the numbers line up and compare +table( + 'Time relative to the fastest in the row', + (median, fastest) => (median / fastest).toFixed(2) + 'x' +); +table('Seconds for one run', median => median.toFixed(3)); diff --git a/test/performance/tools/run.sh b/test/performance/tools/run.sh index f49d4762..6b77b139 100755 --- a/test/performance/tools/run.sh +++ b/test/performance/tools/run.sh @@ -2,21 +2,28 @@ # Compares rr from the working tree, on Node and on Bun, with other tools that replace in files. # Each tool removes the HTML tags from a file in place, on sizes from 1 KB to 100 MB. -# A tool that is not installed is left out. Give sizes as arguments to run only those, like `1KB 10MB`. +# Give sizes as arguments to run only those, like `1KB 10MB`. set -e DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ROOT="$(cd "$DIR/../../.." && pwd)" -SIZES="${*:-1KB 10KB 100KB 1MB 10MB 100MB}" +SIZES="${*:-1KB 10KB 100KB 500KB 1MB 5MB 10MB 50MB 100MB}" +# The file to change is in tmp of the repo, and given by a path from the repo, as rr 3 to 7 can not write to an absolute path outside the working folder +mkdir -p "$ROOT/tmp" TMP="$(mktemp -d)" -trap 'rm -rf "$TMP"' EXIT +WORK="$(mktemp -d "$ROOT/tmp/performance.XXXXXX")" +trap 'rm -rf "$TMP" "$WORK"' EXIT -# The packages are external, so the builds sit next to a node_modules -ln -s "$ROOT/node_modules" "$TMP/node_modules" -(cd "$ROOT" && bun build src/cli.node.ts --target=node --format=esm --packages=external --outfile="$TMP/node.mjs" > /dev/null) -(cd "$ROOT" && bun build src/cli.bun.ts --target=bun --format=esm --packages=external --outfile="$TMP/bun.mjs" > /dev/null) +source "$DIR/../shared/build.sh" +source "$DIR/../shared/check.sh" +build_rr "$ROOT" "$TMP/rr" + +# The tools from npm are the latest, in the temp folder, so nothing is installed for good +npm install --silent --no-audit --no-fund --prefix "$TMP/npm" replace-in-file replace-x replace > /dev/null +NPM="$TMP/npm/node_modules/.bin" +echo "From npm $(cd "$TMP/npm" && npm ls --depth=0 2>/dev/null | tail -n +2 | grep -o '[a-z-]*@[0-9.]*' | tr '\n' ' ')" # The book is 851 KB, so it is repeated to have enough text for the largest file for i in $(seq 130); do cat "$DIR/1984.html"; done > "$TMP/book" @@ -25,7 +32,7 @@ for i in $(seq 130); do cat "$DIR/1984.html"; done > "$TMP/book" # The tools that read the whole file get a pattern that stops at a line break, to do the same work. TAGS='<[^>]*>' TAGS_IN_A_LINE='<[^>\n]*>' -FILE="$TMP/test.file" +FILE="${WORK#"$ROOT/"}/test.file" # BSD sed (macOS) needs an empty suffix after -i, GNU sed does not take one if sed --version > /dev/null 2>&1; then @@ -35,22 +42,28 @@ else fi TOOLS=( - "rr on Node|node $TMP/node.mjs '$TAGS_IN_A_LINE' '' $FILE" - "rr on Bun|bun $TMP/bun.mjs '$TAGS_IN_A_LINE' '' $FILE" + "rr on Node|node $TMP/rr/node.mjs '$TAGS_IN_A_LINE' '' $FILE" + "rr on Bun|bun $TMP/rr/bun.mjs '$TAGS_IN_A_LINE' '' $FILE" "sed|sed $SED_INPLACE 's/$TAGS//g' $FILE" "gsed|gsed -i 's/$TAGS//g' $FILE" "perl|perl -pi -e 's/$TAGS//g' $FILE" "sd|sd '$TAGS_IN_A_LINE' '' $FILE" - "replace-in-file|replace-in-file '/$TAGS_IN_A_LINE/g' '' $FILE --isRegex" + "replace-in-file|$NPM/replace-in-file '/$TAGS_IN_A_LINE/g' '' $FILE --isRegex" + "replace-x|$NPM/replace-x '$TAGS_IN_A_LINE' '' $FILE" + "replace|$NPM/replace '$TAGS_IN_A_LINE' '' $FILE" ) +# A tool that is not installed is left out ARGS=() for tool in "${TOOLS[@]}"; do name="${tool%%|*}" command="${tool#*|}" program="${command%% *}" - command -v "$program" > /dev/null || continue - ARGS+=(-n "$name" "$command") + if command -v "$program" > /dev/null; then + ARGS+=(-n "$name" "$command") + else + echo "Left out $name, as it is not installed" + fi done # The runs start from the repo, a usual place for rr @@ -66,7 +79,13 @@ for size in $SIZES; do esac head -c "$bytes" "$TMP/book" > "$TMP/$size.file" + cp "$TMP/$size.file" "$TMP/$size.reference" + perl -pi -e "s/$TAGS//g" "$TMP/$size.reference" + check_same "$TMP/$size.file" "$FILE" "$TMP/$size.reference" "${ARGS[@]}" + echo '' echo "## $size" - hyperfine -N --warmup 2 --min-runs 10 --prepare "cp $TMP/$size.file $FILE" "${ARGS[@]}" + hyperfine -N --warmup 2 --min-runs 10 --prepare "cp $TMP/$size.file $FILE" --export-json "$TMP/$size.json" "${ARGS[@]}" done + +node "$DIR/../shared/table.js" $(for size in $SIZES; do echo "$TMP/$size.json"; done) diff --git a/test/performance/yargs-vs-argmate/run.sh b/test/performance/yargs-vs-argmate/run.sh index 3fdbd006..cd62c4ed 100755 --- a/test/performance/yargs-vs-argmate/run.sh +++ b/test/performance/yargs-vs-argmate/run.sh @@ -6,7 +6,8 @@ set -e -ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$DIR/../../.." && pwd)" cd "$ROOT" # The last commit with yargs is the parent of the one that took it out of src/cli.ts @@ -15,17 +16,10 @@ BEFORE="${1:-$(git log -1 --format=%h -S"'yargs'" -- src/cli.ts)^}" TMP="$(mktemp -d)" trap 'git -C "$ROOT" worktree remove --force "$TMP/yargs"; rm -rf "$TMP"' EXIT git worktree add --quiet --detach "$TMP/yargs" "$BEFORE" -mkdir "$TMP/argmate" - -# The packages are external, so each build sits next to a node_modules. -# .mjs as the folders have no package.json that says the files are ES modules. -for side in yargs argmate; do - src="$ROOT" - [ "$side" = yargs ] && src="$TMP/yargs" - ln -sfn "$ROOT/node_modules" "$TMP/$side/node_modules" - (cd "$src" && bun build src/cli.node.ts --target=node --format=esm --packages=external --outfile="$TMP/$side/node.mjs" > /dev/null) - (cd "$src" && bun build src/cli.bun.ts --target=bun --format=esm --packages=external --outfile="$TMP/$side/bun.mjs" > /dev/null) -done + +source "$DIR/../shared/build.sh" +build_rr "$TMP/yargs" "$TMP/yargs" +build_rr "$ROOT" "$TMP/argmate" echo 'foobar' > "$TMP/my.file" @@ -36,14 +30,18 @@ cd "$ROOT" echo "yargs from $(git -C "$ROOT" log -1 --format='%h %s' "$BEFORE")" echo 'argmate from the working tree' +ROWS=() compare() { - hyperfine -N --warmup 10 --runs 40 \ - -n "yargs $1" "$2 $TMP/yargs/$3 $4" \ - -n "argmate $1" "$2 $TMP/argmate/$3 $4" + ROWS+=("$TMP/$1.json") + hyperfine -N --warmup 10 --runs 40 --export-json "$TMP/$1.json" \ + -n yargs "$2 $TMP/yargs/$2.mjs $3" \ + -n argmate "$2 $TMP/argmate/$2.mjs $3" } for runtime in node bun; do - compare "$runtime, no flags" "$runtime" "$runtime.mjs" "o x $TMP/my.file -S" - compare "$runtime, many flags" "$runtime" "$runtime.mjs" "o x $TMP/my.file -S -V -m -e utf8 --sort natural -x nope -X nope --name-delimiter , --expect-files" - compare "$runtime, help" "$runtime" "$runtime.mjs" "-h" + compare "$runtime, no flags" "$runtime" "o x $TMP/my.file -S" + compare "$runtime, many flags" "$runtime" "o x $TMP/my.file -S -V -m -e utf8 --sort natural -x nope -X nope --name-delimiter , --expect-files" + compare "$runtime, help" "$runtime" "-h" done + +node "$DIR/../shared/table.js" "${ROWS[@]}" From 41747dd61003b0ceac13e748e88548cc78ccbc30 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Sun, 1 Nov 2026 21:03:20 +1100 Subject: [PATCH 7/9] Time the hotfix releases in readme-speed, add flag-u to see what the flag u costs, and keep a wrong file from a check --- test/performance/README.md | 15 +++++- test/performance/flag-u/run.sh | 68 ++++++++++++++++++++++++++++ test/performance/readme-speed/run.sh | 7 ++- test/performance/shared/check.sh | 4 +- test/performance/tools/run.sh | 2 +- 5 files changed, 90 insertions(+), 6 deletions(-) create mode 100644 test/performance/flag-u/run.sh diff --git a/test/performance/README.md b/test/performance/README.md index 3a322527..5f68af1e 100644 --- a/test/performance/README.md +++ b/test/performance/README.md @@ -6,7 +6,7 @@ Each test has its own folder with a `run.sh`. It builds rr from the working tree `shared` has what the tests have in common, the build of rr (`build.sh`), the check that each tool gives the same file (`check.sh`) and the tables (`table.js`). -Before the timing of each size, every command runs once, and the test stops if one fails or gives another file than the others. A tool that does other work, or no work, would else be timed for it. The file to change is in the ignored `tmp` folder of the repo and is given by a path from the repo, as rr 3 to 7 can not write to an absolute path outside the working folder (#726). They say nothing and end with exit code 0 (#728), so without the check they look fast on big files. +Before the timing of each size, every command runs once, and the test stops if one fails or gives another file than the others. A tool that does other work, or no work, would else be timed for it. The file to change is in the ignored `tmp` folder of the repo and is given by a path from the repo, as rr is most often run on files under the working folder. ## tools @@ -25,7 +25,7 @@ A small file shows the start of each tool, and a big file shows the regex engine ## readme-speed -Runs the speed test of the README again, so its numbers can be checked. rr runs against `sed` on the same sizes as the table there, from 1 KB to 100 MB, with the last release of each major of rr from npm, and with rr from the working tree on Node and on Bun. +Runs the speed test of the README again, so its numbers can be checked. rr runs against `sed` on the same sizes as the table there, from 1 KB to 100 MB, with the last release of each major of rr from npm, and with rr from the working tree on Node and on Bun. rr from the working tree also runs with `-U`, as the flag u is on by default and `-U` turns it off, so the table shows what the flag costs. ```bash bash test/performance/readme-speed/run.sh @@ -36,6 +36,17 @@ The files are made as the README says. Up to 500 KB they are cut from the book, The releases are installed in the temp folder for the run. rr 6 needs `re2`, and the version it asks for does not build on a new Node, so a later `re2` stands in for it. +## flag-u + +Shows what the flag u costs, as it is on by default and `-U` turns it off. rr from the working tree runs with and without `-U`, on Node and on Bun, on files from 1 KB to 100 MB. + +```bash +bash test/performance/flag-u/run.sh +bash test/performance/flag-u/run.sh 1KB 10MB +``` + +Each size runs with two patterns. One removes the HTML tags, so it matches often. The other looks for a tag that ends in `~>`, which is not in the book, so it never matches and the file stays the same. Both try at each `<` and read the tag, so they do the same work, apart from the replacing. + ## yargs-vs-argmate Compares the start of the CLI with yargs against the CLI of the working tree with argmate, on Node and on Bun. Most of a short run is the start, so this is where the parser of the flags shows. diff --git a/test/performance/flag-u/run.sh b/test/performance/flag-u/run.sh new file mode 100644 index 00000000..2eeb4878 --- /dev/null +++ b/test/performance/flag-u/run.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash + +# Shows what the flag u costs, as it is on by default and -U turns it off. +# rr from the working tree runs with and without -U, on Node and on Bun, on short and long files, with a pattern that matches often and one that never matches. +# Give sizes as arguments to run only those, like `1KB 10MB`. + +set -e + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$DIR/../../.." && pwd)" +SIZES="${*:-1KB 100KB 1MB 10MB 100MB}" + +# The file to change is in tmp of the repo, and given by a path from the repo, as rr is most often run on files under the working folder +mkdir -p "$ROOT/tmp" +TMP="$(mktemp -d)" +WORK="$(mktemp -d "$ROOT/tmp/performance.XXXXXX")" +trap 'rm -rf "$TMP" "$WORK"' EXIT + +source "$DIR/../shared/build.sh" +source "$DIR/../shared/check.sh" +build_rr "$ROOT" "$TMP/rr" + +# The book is 851 KB, so it is repeated to have enough text for the largest file +for i in $(seq 130); do cat "$DIR/../tools/1984.html"; done > "$TMP/book" + +# Both patterns try at each < and read the tag, so they do the same work, but ~> is not in the book, so the second never matches +MATCHES='<[^>\n]*>' +NO_MATCHES='<[^>\n]*~>' +FILE="${WORK#"$ROOT/"}/test.file" + +# The runs start from the repo, a usual place for rr +cd "$ROOT" + +ROWS=() +for size in $SIZES; do + number="${size//[!0-9]/}" + unit="${size//[0-9]/}" + case "$unit" in + KB) bytes=$((number * 1024)) ;; + MB) bytes=$((number * 1024 * 1024)) ;; + *) echo "Unknown size $size, use KB or MB like 10KB" && exit 1 ;; + esac + head -c "$bytes" "$TMP/book" > "$TMP/$size.file" + + cp "$TMP/$size.file" "$TMP/$size.matches" + perl -pi -e 's/<[^>]*>//g' "$TMP/$size.matches" + cp "$TMP/$size.file" "$TMP/$size.no-matches" + + for kind in matches no-matches; do + pattern="$MATCHES" + [ "$kind" = no-matches ] && pattern="$NO_MATCHES" + ARGS=( + -n 'Node' "node $TMP/rr/node.mjs '$pattern' '' $FILE" + -n 'Node -U' "node $TMP/rr/node.mjs '$pattern' '' $FILE -U" + -n 'Bun' "bun $TMP/rr/bun.mjs '$pattern' '' $FILE" + -n 'Bun -U' "bun $TMP/rr/bun.mjs '$pattern' '' $FILE -U" + ) + check_same "$TMP/$size.file" "$FILE" "$TMP/$size.$kind" "${ARGS[@]}" + + row="$size $kind" + echo '' + echo "## $row" + hyperfine -N --warmup 2 --min-runs 10 --prepare "cp $TMP/$size.file $FILE" --export-json "$TMP/$row.json" "${ARGS[@]}" + ROWS+=("$TMP/$row.json") + done +done + +node "$DIR/../shared/table.js" "${ROWS[@]}" diff --git a/test/performance/readme-speed/run.sh b/test/performance/readme-speed/run.sh index 8de551c0..45f30dbd 100755 --- a/test/performance/readme-speed/run.sh +++ b/test/performance/readme-speed/run.sh @@ -9,9 +9,9 @@ set -e DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ROOT="$(cd "$DIR/../../.." && pwd)" SIZES="${*:-1KB 5KB 10KB 100KB 500KB 1MB 5MB 10MB 25MB 50MB 100MB}" -VERSIONS='1.1.0 2.5.3 3.1.1 4.1.2 5.3.0 6.1.5 7.1.15' +VERSIONS='1.1.1 2.5.4 3.1.2 4.1.3 5.3.1 6.1.6 7.1.16' -# The file to change is in tmp of the repo, and given by a path from the repo, as rr 3 to 7 can not write to an absolute path outside the working folder +# The file to change is in tmp of the repo, and given by a path from the repo, as rr is most often run on files under the working folder mkdir -p "$ROOT/tmp" TMP="$(mktemp -d)" WORK="$(mktemp -d "$ROOT/tmp/performance.XXXXXX")" @@ -44,8 +44,11 @@ ARGS=(-n 'sed' "sed $SED_INPLACE 's/$TAGS//g' $FILE") for version in $VERSIONS; do ARGS+=(-n "rr $version" "node $TMP/$version/node_modules/.bin/rexreplace '$TAGS_IN_A_LINE' '' $FILE") done +# The flag u is on by default now, and -U turns it off, to see what it costs ARGS+=(-n 'rr now' "node $TMP/rr/node.mjs '$TAGS_IN_A_LINE' '' $FILE") +ARGS+=(-n 'rr now -U' "node $TMP/rr/node.mjs '$TAGS_IN_A_LINE' '' $FILE -U") ARGS+=(-n 'rr now on Bun' "bun $TMP/rr/bun.mjs '$TAGS_IN_A_LINE' '' $FILE") +ARGS+=(-n 'rr now on Bun -U' "bun $TMP/rr/bun.mjs '$TAGS_IN_A_LINE' '' $FILE -U") head -c $((500 * 1024)) "$DIR/../tools/1984.html" > "$TMP/500KB.part" diff --git a/test/performance/shared/check.sh b/test/performance/shared/check.sh index f144df37..1665b8c5 100644 --- a/test/performance/shared/check.sh +++ b/test/performance/shared/check.sh @@ -11,8 +11,10 @@ check_same() { if ! eval "$command" > /dev/null 2>&1 < /dev/null; then echo "$name failed" && exit 1 fi + # The wrong file is kept in tmp of the repo, to see what went wrong if ! cmp -s "$file" "$reference"; then - echo "$name does not give the same file as the others" && exit 1 + cp "$file" "$ROOT/tmp/$name.wrong" + echo "$name does not give the same file as the others, see tmp/$name.wrong" && exit 1 fi done } diff --git a/test/performance/tools/run.sh b/test/performance/tools/run.sh index 6b77b139..9efa3d50 100755 --- a/test/performance/tools/run.sh +++ b/test/performance/tools/run.sh @@ -10,7 +10,7 @@ DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ROOT="$(cd "$DIR/../../.." && pwd)" SIZES="${*:-1KB 10KB 100KB 500KB 1MB 5MB 10MB 50MB 100MB}" -# The file to change is in tmp of the repo, and given by a path from the repo, as rr 3 to 7 can not write to an absolute path outside the working folder +# The file to change is in tmp of the repo, and given by a path from the repo, as rr is most often run on files under the working folder mkdir -p "$ROOT/tmp" TMP="$(mktemp -d)" WORK="$(mktemp -d "$ROOT/tmp/performance.XXXXXX")" From 4c998026f19feee19ee11065ff6dc924b752e6f1 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Sun, 1 Nov 2026 21:28:54 +1100 Subject: [PATCH 8/9] Turn flag-u into engines, to compare rr, runtime and regex+ with the flags u, v and none --- test/performance/README.md | 10 +++--- test/performance/{flag-u => engines}/run.sh | 36 +++++++++++++-------- 2 files changed, 28 insertions(+), 18 deletions(-) rename test/performance/{flag-u => engines}/run.sh (61%) diff --git a/test/performance/README.md b/test/performance/README.md index 5f68af1e..e0e5ab8d 100644 --- a/test/performance/README.md +++ b/test/performance/README.md @@ -36,15 +36,17 @@ The files are made as the README says. Up to 500 KB they are cut from the book, The releases are installed in the temp folder for the run. rr 6 needs `re2`, and the version it asks for does not build on a new Node, so a later `re2` stands in for it. -## flag-u +## engines -Shows what the flag u costs, as it is on by default and `-U` turns it off. rr from the working tree runs with and without `-U`, on Node and on Bun, on files from 1 KB to 100 MB. +Compares the regex engines of rr, and what the flag u costs, as it is on by default. rr from the working tree runs on Node and on Bun, on files from 1 KB to 100 MB. ```bash -bash test/performance/flag-u/run.sh -bash test/performance/flag-u/run.sh 1KB 10MB +bash test/performance/engines/run.sh +bash test/performance/engines/run.sh 1KB 10MB ``` +The engines are `rr`, the default, `runtime`, the native engine of the runtime, and `regex+`, which uses the flag v. `rr` and `runtime` each run with the default flag u, with `-U`, and with the flag v in place of u. Each row is one runtime, so the 1.00x is the fastest engine on that runtime. + Each size runs with two patterns. One removes the HTML tags, so it matches often. The other looks for a tag that ends in `~>`, which is not in the book, so it never matches and the file stays the same. Both try at each `<` and read the tag, so they do the same work, apart from the replacing. ## yargs-vs-argmate diff --git a/test/performance/flag-u/run.sh b/test/performance/engines/run.sh similarity index 61% rename from test/performance/flag-u/run.sh rename to test/performance/engines/run.sh index 2eeb4878..c4853f39 100644 --- a/test/performance/flag-u/run.sh +++ b/test/performance/engines/run.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash -# Shows what the flag u costs, as it is on by default and -U turns it off. -# rr from the working tree runs with and without -U, on Node and on Bun, on short and long files, with a pattern that matches often and one that never matches. +# Compares the regex engines of rr, and the flag u that is on by default against -U and the flag v. +# rr from the working tree runs on Node and on Bun, on short and long files, with a pattern that matches often and one that never matches. # Give sizes as arguments to run only those, like `1KB 10MB`. set -e @@ -49,19 +49,27 @@ for size in $SIZES; do for kind in matches no-matches; do pattern="$MATCHES" [ "$kind" = no-matches ] && pattern="$NO_MATCHES" - ARGS=( - -n 'Node' "node $TMP/rr/node.mjs '$pattern' '' $FILE" - -n 'Node -U' "node $TMP/rr/node.mjs '$pattern' '' $FILE -U" - -n 'Bun' "bun $TMP/rr/bun.mjs '$pattern' '' $FILE" - -n 'Bun -U' "bun $TMP/rr/bun.mjs '$pattern' '' $FILE -U" - ) - check_same "$TMP/$size.file" "$FILE" "$TMP/$size.$kind" "${ARGS[@]}" + for runtime in Node Bun; do + rr="node $TMP/rr/node.mjs '$pattern' '' $FILE" + [ "$runtime" = Bun ] && rr="bun $TMP/rr/bun.mjs '$pattern' '' $FILE" + # The default flags are gimu, so --flags gimv is the default with v in place of u + ARGS=( + -n 'rr' "$rr" + -n 'rr -U' "$rr -U" + -n 'rr v' "$rr --flags gimv" + -n 'runtime' "$rr -E runtime" + -n 'runtime -U' "$rr -E runtime -U" + -n 'runtime v' "$rr -E runtime --flags gimv" + -n 'regex+' "$rr -E regex+" + ) + check_same "$TMP/$size.file" "$FILE" "$TMP/$size.$kind" "${ARGS[@]}" - row="$size $kind" - echo '' - echo "## $row" - hyperfine -N --warmup 2 --min-runs 10 --prepare "cp $TMP/$size.file $FILE" --export-json "$TMP/$row.json" "${ARGS[@]}" - ROWS+=("$TMP/$row.json") + row="$size $kind on $runtime" + echo '' + echo "## $row" + hyperfine -N --warmup 2 --min-runs 10 --prepare "cp $TMP/$size.file $FILE" --export-json "$TMP/$row.json" "${ARGS[@]}" + ROWS+=("$TMP/$row.json") + done done done From 8dcf1976f7016c62ae56f4b3d13d5726d76d6649 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Mon, 2 Nov 2026 09:24:16 +1100 Subject: [PATCH 9/9] Order the engines test by flag --- test/performance/engines/results-nov-2026.md | 50 ++++++++++++++++++++ test/performance/engines/run.sh | 6 +-- 2 files changed, 53 insertions(+), 3 deletions(-) create mode 100644 test/performance/engines/results-nov-2026.md diff --git a/test/performance/engines/results-nov-2026.md b/test/performance/engines/results-nov-2026.md new file mode 100644 index 00000000..f9e9aeb6 --- /dev/null +++ b/test/performance/engines/results-nov-2026.md @@ -0,0 +1,50 @@ +Time relative to the fastest in the row + +| | rr | rr -U | rr v | runtime | runtime -U | runtime v | regex+ | +|---|---:|---:|---:|---:|---:|---:|---:| +| 1KB matches on Node | 1.03x | 1.03x | 1.02x | 1.00x | 1.00x | 1.01x | 1.06x | +| 1KB matches on Bun | 1.03x | 1.02x | 1.00x | 1.00x | 1.02x | 1.00x | 1.08x | +| 1KB no-matches on Node | 1.00x | 1.01x | 1.01x | 1.02x | 1.00x | 1.01x | 1.02x | +| 1KB no-matches on Bun | 1.02x | 1.01x | 1.01x | 1.00x | 1.00x | 1.01x | 1.03x | +| 100KB matches on Node | 1.00x | 1.02x | 1.01x | 1.01x | 1.02x | 1.00x | 1.02x | +| 100KB matches on Bun | 1.02x | 1.00x | 1.01x | 1.01x | 1.00x | 1.06x | 1.06x | +| 100KB no-matches on Node | 1.00x | 1.00x | 1.01x | 1.02x | 1.03x | 1.01x | 1.03x | +| 100KB no-matches on Bun | 1.01x | 1.05x | 1.04x | 1.01x | 1.00x | 1.01x | 1.03x | +| 1MB matches on Node | 1.02x | 1.00x | 1.01x | 1.01x | 1.03x | 1.00x | 1.04x | +| 1MB matches on Bun | 1.05x | 1.03x | 1.06x | 1.00x | 1.00x | 1.02x | 1.04x | +| 1MB no-matches on Node | 1.01x | 1.00x | 1.02x | 1.02x | 1.04x | 1.02x | 1.07x | +| 1MB no-matches on Bun | 1.02x | 1.00x | 1.03x | 1.02x | 1.00x | 1.02x | 1.05x | +| 10MB matches on Node | 1.06x | 1.00x | 1.05x | 1.04x | 1.01x | 1.02x | 1.07x | +| 10MB matches on Bun | 1.05x | 1.02x | 1.04x | 1.04x | 1.00x | 1.02x | 1.04x | +| 10MB no-matches on Node | 1.12x | 1.00x | 1.13x | 1.10x | 1.01x | 1.12x | 1.12x | +| 10MB no-matches on Bun | 1.12x | 1.02x | 1.14x | 1.22x | 1.00x | 1.13x | 1.14x | +| 100MB matches on Node | 1.09x | 1.00x | 1.13x | 1.08x | 1.01x | 1.11x | 1.09x | +| 100MB matches on Bun | 1.19x | 1.00x | 1.15x | 1.13x | 1.01x | 1.20x | 1.10x | +| 100MB no-matches on Node | 1.30x | 1.00x | 1.30x | 1.30x | 1.01x | 1.30x | 1.32x | +| 100MB no-matches on Bun | 1.33x | 1.00x | 1.39x | 1.32x | 1.01x | 1.33x | 1.40x | + +Seconds for one run + +| | rr | rr -U | rr v | runtime | runtime -U | runtime v | regex+ | +|---|---:|---:|---:|---:|---:|---:|---:| +| 1KB matches on Node | 0.073 | 0.073 | 0.073 | 0.071 | 0.072 | 0.072 | 0.075 | +| 1KB matches on Bun | 0.037 | 0.037 | 0.037 | 0.037 | 0.037 | 0.036 | 0.040 | +| 1KB no-matches on Node | 0.070 | 0.071 | 0.071 | 0.072 | 0.070 | 0.071 | 0.072 | +| 1KB no-matches on Bun | 0.037 | 0.037 | 0.036 | 0.036 | 0.036 | 0.037 | 0.037 | +| 100KB matches on Node | 0.073 | 0.074 | 0.074 | 0.074 | 0.074 | 0.073 | 0.074 | +| 100KB matches on Bun | 0.037 | 0.037 | 0.037 | 0.037 | 0.037 | 0.039 | 0.039 | +| 100KB no-matches on Node | 0.071 | 0.071 | 0.072 | 0.073 | 0.073 | 0.072 | 0.073 | +| 100KB no-matches on Bun | 0.037 | 0.038 | 0.038 | 0.037 | 0.036 | 0.037 | 0.038 | +| 1MB matches on Node | 0.079 | 0.078 | 0.079 | 0.079 | 0.081 | 0.078 | 0.081 | +| 1MB matches on Bun | 0.040 | 0.039 | 0.040 | 0.038 | 0.038 | 0.039 | 0.040 | +| 1MB no-matches on Node | 0.074 | 0.073 | 0.075 | 0.075 | 0.076 | 0.075 | 0.078 | +| 1MB no-matches on Bun | 0.038 | 0.037 | 0.038 | 0.038 | 0.037 | 0.038 | 0.039 | +| 10MB matches on Node | 0.133 | 0.125 | 0.131 | 0.130 | 0.127 | 0.128 | 0.135 | +| 10MB matches on Bun | 0.063 | 0.061 | 0.062 | 0.062 | 0.060 | 0.061 | 0.062 | +| 10MB no-matches on Node | 0.115 | 0.102 | 0.115 | 0.113 | 0.103 | 0.114 | 0.115 | +| 10MB no-matches on Bun | 0.059 | 0.054 | 0.060 | 0.064 | 0.052 | 0.059 | 0.060 | +| 100MB matches on Node | 0.649 | 0.598 | 0.676 | 0.644 | 0.607 | 0.663 | 0.650 | +| 100MB matches on Bun | 0.292 | 0.246 | 0.284 | 0.278 | 0.249 | 0.296 | 0.271 | +| 100MB no-matches on Node | 0.506 | 0.389 | 0.505 | 0.505 | 0.392 | 0.506 | 0.514 | +| 100MB no-matches on Bun | 0.256 | 0.192 | 0.267 | 0.254 | 0.194 | 0.256 | 0.268 | + diff --git a/test/performance/engines/run.sh b/test/performance/engines/run.sh index c4853f39..92ce4411 100644 --- a/test/performance/engines/run.sh +++ b/test/performance/engines/run.sh @@ -52,13 +52,13 @@ for size in $SIZES; do for runtime in Node Bun; do rr="node $TMP/rr/node.mjs '$pattern' '' $FILE" [ "$runtime" = Bun ] && rr="bun $TMP/rr/bun.mjs '$pattern' '' $FILE" - # The default flags are gimu, so --flags gimv is the default with v in place of u + # Grouped by flag, as the flag is what moves the time. The default flags are gimu, so --flags gimv is the default with v in place of u ARGS=( -n 'rr' "$rr" - -n 'rr -U' "$rr -U" - -n 'rr v' "$rr --flags gimv" -n 'runtime' "$rr -E runtime" + -n 'rr -U' "$rr -U" -n 'runtime -U' "$rr -E runtime -U" + -n 'rr v' "$rr --flags gimv" -n 'runtime v' "$rr -E runtime --flags gimv" -n 'regex+' "$rr -E regex+" )