|
| 1 | +# P-026 — Naughty-strings robustness pack (BLNS-driven crash testing) |
| 2 | + |
| 3 | +- **Status:** draft |
| 4 | +- **Depends on / relates to:** |
| 5 | + - [P-001](P-001-csharp-extractor.md) — the C# → OwnIR extractor: the thing that |
| 6 | + has to survive arbitrary third-party source text in the first place. |
| 7 | + - [P-012](P-012-bug-corpus-mining.md) — same "curated corpus, gated in CI" |
| 8 | + shape, but keyed by **string content**, not by bug pattern; orthogonal, not a |
| 9 | + replacement. |
| 10 | + - [P-015](P-015-configuration-surface.md) — the future `own.toml`/`.ownrc` |
| 11 | + config surface; config discovery has to survive naughty paths/globs too. |
| 12 | + - [P-024](P-024-security-audit-profile.md) — same "берём готовое, не |
| 13 | + изобретаем свою эвристику" instinct (adopt an existing corpus instead of |
| 14 | + hand-rolling a dozen Unicode edge cases), but explicitly **not** a security |
| 15 | + profile — see Non-goals. |
| 16 | + |
| 17 | +## Motivation |
| 18 | + |
| 19 | +Own.NET's whole value proposition is running against real, uncontrolled legacy |
| 20 | +C#/WPF/DevExpress code: arbitrary identifiers, string literals, resource |
| 21 | +strings, file and project paths chosen by other people over twenty years. The |
| 22 | +project's own honest-skip philosophy (`docs/ROADMAP.md`) already treats "the |
| 23 | +checker doesn't know" as an acceptable, first-class outcome — but a **crash** |
| 24 | +is not "doesn't know," it's the tool falling over on a customer's codebase, |
| 25 | +which is strictly worse than a missed diagnostic. |
| 26 | + |
| 27 | +Today robustness is exercised only by *valid* fixtures — `corpus/wpf/`, |
| 28 | +`corpus/real-world/`, `tests/fixtures/` — plus whatever `ParseError`/`LexError` |
| 29 | +paths (`ownlang/lexer.py`, `ownlang/parser.py`) happen to be hit incidentally. |
| 30 | +Nothing in the suite deliberately throws adversarial *text* at the extractor, |
| 31 | +the JSON/SARIF emitters (`ownlang/cfg_json.py`, `ownlang/diag_sarif.py`, |
| 32 | +`ownlang/diagnostics.py`, `ownlang/report.py`), or the CLI (`ownlang/__main__.py`) |
| 33 | +— zero-width joiners, RTL/LTR override characters, unpaired surrogates, SQL/ |
| 34 | +XSS-shaped strings sitting inert inside a C# string literal, absurdly long |
| 35 | +lines, mixed line endings, strings that are themselves valid-looking JSON or |
| 36 | +XML, "your kernel just crashed"-style command injection payloads. And this |
| 37 | +class of bug is not hypothetical here: `OwnAudit/Run-Audit.ps1` already carries |
| 38 | +a scar from exactly this — `PYTHONUTF8=1` is set specifically to dodge a |
| 39 | +**cp1251 console crash on a Russian-locale Windows target**. That is one |
| 40 | +instance of the bug class BLNS exists to catch *systematically*, found the hard |
| 41 | +way instead of by test. |
| 42 | + |
| 43 | +[Big List of Naughty Strings](https://github.com/minimaxir/big-list-of-naughty-strings) |
| 44 | +(BLNS) is a maintained, MIT-licensed corpus built for exactly this: ~500 strings |
| 45 | +(Unicode edge cases, escaping/injection-shaped strings, whitespace and |
| 46 | +line-ending oddities, format-breakers for JSON/XML/CSV/SQL/shell), shipped as a |
| 47 | +plain `blns.json` array plus a `.NET` port (`NaughtyStrings` NuGet package) for |
| 48 | +the C#-side pieces (`audit/`'s eventual C# on lift-out, per |
| 49 | +[`OwnAudit/README.md`](https://github.com/PhysShell/OwnAudit)). BLNS itself is |
| 50 | +explicit that it is not a substitute for real security testing (see |
| 51 | +Non-goals) — its contract here is narrower and cheaper: **the tool must not |
| 52 | +crash, hang, or corrupt output on any string in the corpus.** |
| 53 | + |
| 54 | +## Scope |
| 55 | + |
| 56 | +1. **Vendor the corpus.** A pinned, static copy of `blns.json` (upstream tag/ |
| 57 | + commit recorded in a comment) as a fixture, e.g. |
| 58 | + `tests/fixtures/blns.json` — no network fetch at test time, no submodule |
| 59 | + (matches the project's existing "no external runtime deps beyond stdlib" |
| 60 | + posture in the Python core). |
| 61 | + |
| 62 | +2. **Layer 1 — lexer/parser/extractor.** Parametrize over every BLNS entry, |
| 63 | + embedding it as: (a) `.own` string-literal content, (b) a C# string literal |
| 64 | + fed through the P-001 extractor, (c) a file/module name passed on the CLI. |
| 65 | + Assert only: no unhandled exception escapes `ownlang/lexer.py` / |
| 66 | + `ownlang/parser.py` / the extractor; the *only* acceptable failure shapes |
| 67 | + are `LexError`/`ParseError` (or the extractor's own diagnostic-and-skip |
| 68 | + path) — never a raw traceback, never a hang past a fixed timeout. |
| 69 | + |
| 70 | +3. **Layer 2 — serialization.** Pipe BLNS content through |
| 71 | + `ownlang/diagnostics.py` → `ownlang/diag_sarif.py` / `ownlang/cfg_json.py` / |
| 72 | + `ownlang/report.py` (as a synthesized finding message / file path / symbol |
| 73 | + name) and assert the emitted JSON/SARIF/Markdown is well-formed |
| 74 | + (round-trips through a JSON/SARIF parser) with no crash — this is the |
| 75 | + layer `test_cfg_json.py` / `test_diag_sarif.py` already exercise for valid |
| 76 | + input; this proposal is the adversarial-input twin. |
| 77 | + |
| 78 | +4. **Layer 3 — CLI & future config.** `ownlang/__main__.py` argument/path |
| 79 | + handling, and (when [P-015](P-015-configuration-surface.md) lands) `own.toml` |
| 80 | + discovery, given BLNS-flavored file names, directory names, and glob |
| 81 | + patterns. |
| 82 | + |
| 83 | +5. **Land as one hermetic, parametrized module** — |
| 84 | + `tests/test_naughty_strings.py` — wired into `tests/run_tests.py` and CI the |
| 85 | + same way `tests/test_corpus.py` is: fast, offline, property-style |
| 86 | + ("must not crash," not "must produce code X"). |
| 87 | + |
| 88 | +6. **Follow-on, not in v0:** an equivalent pass over `OwnAudit`'s SARIF |
| 89 | + ingestion / `artifacts/health-report.*` rendering, since that's the other |
| 90 | + place free text from arbitrary source flows into output — deferred because |
| 91 | + it crosses the repo boundary and OwnAudit already treats SARIF as its |
| 92 | + external contract. |
| 93 | + |
| 94 | +## Non-goals |
| 95 | + |
| 96 | +- **Not a security test / pentest substitute.** BLNS's own README says the |
| 97 | + same. This proposal claims only "does not crash / does not corrupt state on |
| 98 | + adversarial text" — nothing about exploitability, authorization, or network |
| 99 | + surface. That territory is [P-024](P-024-security-audit-profile.md)'s, and |
| 100 | + this proposal does not overlap it: no scanning, no CVE claims, no new |
| 101 | + security-flavored diagnostic codes. |
| 102 | +- **Not a new checker or diagnostic.** No new `OWN0NN` code, no severity |
| 103 | + change, no touch to ownership/lifetime semantics. Purely a regression/ |
| 104 | + robustness harness around existing entry points. |
| 105 | +- **Not coverage-guided fuzzing.** That is `007`'s `fuzz/` (cargo-fuzz) |
| 106 | + territory on the eventual Rust core (P-022) — an open-ended search for novel |
| 107 | + crashes. This is a fixed, curated, deterministic corpus, cheap enough to run |
| 108 | + on every commit, not a campaign. |
| 109 | +- **Not "every naughty string gets a pretty diagnostic."** The honest-skip / |
| 110 | + `ParseError` contract is sufficient; the property under test is "no crash, |
| 111 | + no hang, no corrupted output," not "graceful handling with a nice message" |
| 112 | + for all ~500 entries. |
| 113 | +- **Does not change the `corpus/` layout** (`before.cs`/`after.cs`/`case.own`) |
| 114 | + from P-012 — BLNS fixtures are a separate, orthogonal corpus keyed by string |
| 115 | + content, not by bug pattern, and live under `tests/fixtures/`, not `corpus/`. |
| 116 | + |
| 117 | +## Sketch |
| 118 | + |
| 119 | +```text |
| 120 | +tests/fixtures/blns.json # vendored, pinned copy (upstream commit noted) |
| 121 | +tests/test_naughty_strings.py # parametrized over every entry, 3 layers above |
| 122 | +``` |
| 123 | + |
| 124 | +```python |
| 125 | +import json, os, pytest |
| 126 | +from ownlang.lexer import LexError |
| 127 | +from ownlang.parser import ParseError, parse |
| 128 | + |
| 129 | +with open(os.path.join(os.path.dirname(__file__), "fixtures", "blns.json"), |
| 130 | + encoding="utf-8") as f: |
| 131 | + BLNS = json.load(f) |
| 132 | + |
| 133 | +@pytest.mark.parametrize("naughty", BLNS) |
| 134 | +def test_parser_does_not_crash(naughty): |
| 135 | + src = f'resource R;\nfn f() {{ let s = "{naughty}"; }}\n' |
| 136 | + try: |
| 137 | + parse(src) |
| 138 | + except (ParseError, LexError): |
| 139 | + pass # an honest rejection is fine; anything else is a bug |
| 140 | +``` |
| 141 | + |
| 142 | +Serialization side follows the same shape against `diag_sarif.py` / |
| 143 | +`cfg_json.py`, asserting `json.loads(...)` / a SARIF-shape check succeeds. |
| 144 | + |
| 145 | +## Open questions |
| 146 | + |
| 147 | +1. **Generation vs. fixture files.** Synthesize `.own`/`.cs` source around each |
| 148 | + BLNS entry on the fly (parametrized, no repo bloat — the sketch above) vs. |
| 149 | + materializing ~500 tiny fixture files. Leaning generation; only fall back to |
| 150 | + files if a specific entry needs a shape the generator can't express. |
| 151 | +2. **Timeout bound.** Several BLNS entries are specifically shaped to blow up |
| 152 | + naive parsers (repetition/expansion strings). What per-case wall-clock |
| 153 | + cutoff counts as "hung" in CI? |
| 154 | +3. **Encoding boundary.** Is UTF-8 the only contract for this pack, or does |
| 155 | + OwnAudit's cp1251-console incident warrant its own explicit BLNS pass over |
| 156 | + the PowerShell/console path, given it already burned once? |
| 157 | +4. **Vendoring mechanics.** Pinned static copy of `blns.json` (simple, matches |
| 158 | + current no-submodule posture) vs. a `scripts/` updater that re-fetches on |
| 159 | + demand — leaning static copy with the upstream commit noted in a header |
| 160 | + comment. |
| 161 | +5. **Rust-core inheritance.** When/if P-022's Rust core lands with a |
| 162 | + differential oracle (Python = golden), does this pack become a shared input |
| 163 | + fed to both sides rather than a Python-only test? |
| 164 | +6. **OwnAudit follow-on timing.** Scope item 6 defers the OwnAudit-side pass — |
| 165 | + confirm that's the right call now vs. folding it in immediately given the |
| 166 | + cp1251 precedent already lives there. |
0 commit comments