🛡️ PySanitize

Multi-format document desensitization · local parsing · rules + LLM locating · class- and text-driven image mosaicing · layout-preserving PDF redaction

PDF · DOCX · Excel · scans → sanitized Markdown + mosaiced images + audit report + redacted PDF

English | 中文

PySanitize desensitizes sensitive information in PDF / DOCX / Excel / scanned documents, fully local parsing on top of MinerU — no cloud calls.

✨ Features


🧩 Multi-format input	PDF / image / DOCX / PPTX / XLSX parsed into one unified structure
🔍 Rules + LLM dual engine	regex + dictionary heuristics (offline-ready); the LLM only locates, never rewrites, with verbatim re-match against hallucination
🖼️ Image masking	`face` / any YOLO class (detection models) and text-driven — bare `--image-text` mosaics all printed text, or a field list mosaics only matching fields (e.g. a company name on a seal) → mosaic
📄 PDF redaction	`--redact-pdf` also yields a layout-preserving `redacted.pdf` — sensitive glyphs truly deleted, mosaic on top, table borders intact
🔁 Reversible masking	`--recoverable` records each value's ciphertext in `audit.json`; `pysanitize --recover` restores the original with the passphrase — the sanitized document itself looks like any normal run
🔌 Switchable providers	`--provider openai \| claude \| openrouter`, pick a provider section per model yaml
📊 Audit-friendly	public summary has no raw values; the raw-value report is only written with `--audit`
🛡️ Fault-tolerant	missing keys / optional deps / model downloads degrade with a warning, never a hard crash

🔄 Pipeline

input(PDF/DOCX/Excel/scan)
  → [parser]      MinerU middle.json (per-line geometry) → ParsedDocument{text, blocks, images/}
  → [detector]    rules + LLM locate sensitive fields → exact char offsets
  → [masker]      mask by field type (138****5678 / **** / keep head-tail N)
  → [image]       class- (face/YOLO) + text-driven (OCR → all text or matching fields) → PIL mosaic
  → [redact]      (PDF only, opt-in) offsets → page rects → redacted.pdf (glyphs deleted, mosaic)
  → output        sanitized.md + images_masked/ + audit.json
                  (--recoverable: audit.json also carries the ciphertext → restorable with --recover)

📦 Installation

Requires Python ≥ 3.12; uv recommended:

git clone https://github.com/LyoAI/PySanitize.git && cd PySanitize
uv sync                    # all deps (incl. mineru[pipeline], large)

config/ and .env are git-ignored and optional — the tool ships built-in defaults (field specs, pipeline switches), so a fresh clone and CI work out of the box. To customize or add real keys, create them locally:

config/fields.yaml, config/pipeline.yaml, config/llm/<model>.yaml — override the built-in defaults (see the config sections below; the default field specs live in pysanitize/detector/specs.py).
.env — API keys referenced by config/llm/*.yaml ${VAR} placeholders:

DEEPSEEK_API_KEY=...      # deepseek-v4-flash (default model)
DASHSCOPE_API_KEY=...     # qwen3-max / qwen3.6-27b
MINERU_BACKEND=pipeline   # pipeline (CPU) | vlm-engine / hybrid-engine (GPU)
MODELS_DIR=~/Models       # downloaded local models (YuNet ONNX, ...)
LLM_TIMEOUT_S=180         # per-call LLM request timeout (seconds)
PYSANITIZE_RECOVER_KEY=... # --recoverable / --recover passphrase (optional; else --recover-key / .recover.key)

Optional features, unlock on demand:

uv sync --extra image-yolo      # YOLO general object detection (ultralytics)
uv sync --extra image-ocr       # OCR text-region detection (paddleocr, large)
uv sync --extra tui             # interactive TUI frontend (textual)
uv sync --extra recover         # reversible masking (--recoverable / --recover; cryptography)

🚀 Quick start

# 1) Pure local rules detection (offline, no LLM)
uv run pysanitize sample.pdf

# 2) Hybrid: rules + LLM locating (default openai/deepseek-v4-flash)
uv run pysanitize sample.pdf --detector hybrid

#    pick the LLM provider + model (provider = a section in the model's yaml)
uv run pysanitize sample.pdf --detector hybrid --provider openai    --model deepseek-v4-flash
uv run pysanitize sample.pdf --detector llm     --provider openrouter --model deepseek-v4-flash

# 3) Image masking: no image is detected by default — name your targets explicitly
uv run pysanitize sample.pdf --mask-images --image-classes face   # faces
uv run pysanitize sample.pdf --mask-images --image-text         # all printed text (needs --extra image-ocr)

# 4) Restrict fields + write a raw-value audit report (local review only, do not share)
uv run pysanitize sample.xlsx --fields person_name,phone --audit

# 5) Layout-preserving PDF redaction (opt-in for PDF inputs)
uv run pysanitize sample.pdf --redact-pdf                     # writes output/<stem>/redacted.pdf
uv run pysanitize sample.pdf --redact-pdf --redaction-style block   # solid box instead of mosaic

# 6) Field-driven image masking: OCR the images, mask only the matching fields
uv run pysanitize sample.pdf --mask-images --image-text company_name,address

# 7) Reversible masking + restore (needs --extra recover)
uv run pysanitize sample.pdf --recoverable
#    → output looks like a normal run; audit.json records each value's ciphertext
uv run pysanitize output/sample/sanitized.md --recover          # md restores exactly
uv run pysanitize output/sample/redacted.pdf --recover          # pdf: values back, layout approximate

Each run produces a job directory (default output/<doc-name>/):

output/<doc-name>/
├── sanitized.md            # sanitized Markdown (image links point into images_masked/)
├── images_masked/          # every extracted image — masked copies, or originals when masking is off
├── redacted.pdf            # PDF inputs with --redact-pdf: original layout, regions deleted + mosaiced
├── audit.json              # public summary: hit counts + masked text, no raw values
│                             with --recoverable: also each span's ciphertext + position
└── .recover.key            # with --recoverable: the effective passphrase (0600), always materialized

With --audit, an extra sensitive_report.json is written (raw values + char offsets, for local review — do not share). Any flag you don't pass falls back to config/pipeline.yaml; explicit flags win.

🖥️ Interactive TUI

uv sync --extra tui        # one-time: installs Textual
uv run pysanitize --launch tui

A six-tab terminal UI (Textual): Fields — checkbox-select the sensitive field types from config/fields.yaml; Options — input file, detection mode, LLM endpoint, output, plus the Recoverable switch and a password-masked Recovery key (blank = env / .recover.key / auto-generate); ③ Image — image masking targets (enable, class list, all-text switch, detector) plus a "Same as text" toggle that lets you pick a different (possibly larger) field set to OCR inside images; Run — type free-form requirements that are appended to the LLM prompt, then run with a live log; Results — per-field hit counts and output paths; ⑥ Recover — point at a sanitized.md / redacted.pdf (its audit.json beside it), optionally type the passphrase, and restore the original in place of the CLI's --recover. Quit with the ✕ button or ctrl+c (the default ctrl+q is swallowed by some terminals, and cmd+q belongs to macOS). The plain CLI stays the primary interface — the TUI is a convenience layer over the same pipeline (pysanitize.core.run_sanitizer).

🐍 Python API

from pysanitize.pipeline import sanitize_document

result = sanitize_document(
    "contract.pdf",
    detector="hybrid",          # rules | llm | hybrid
    llm_model="deepseek-v4-flash",  # config/llm/<model>.yaml filename
    llm_provider="openai",      # provider section in that yaml: openai | claude | openrouter
    fields=["phone", "company_name", "person_name"],
    mask_images=True,
    image_classes=["face"],     # image targets: face | text | <YOLO class>; empty = none
    image_fields=["company_name", "address"],  # None = same as fields; [] = no image-field masking
    redact_pdf=True,            # opt-in: also write redacted.pdf for PDF sources
    redaction_style="mosaic",   # mosaic | block
    audit=False,
    recoverable=True,           # audit.json records ciphertext for --recover (needs the recover extra)
    recover_key="passphrase",   # else $PYSANITIZE_RECOVER_KEY, else generated — effective key → .recover.key
)
print(result.sanitized_md)     # Path
print(result.redacted_pdf)     # Path | None (PDF sources with resolvable regions)
print(result.detections)       # each with field_type / start / end / masked_value

🔍 Detection modes

mode	description
`rules`	pure-local regex + dictionary heuristics (ID/USCC checksums, surname dict, company suffix dict), offline-ready
`llm`	LLM locates spans per chunk: returns `{"field_type", "value"}`, value must be a verbatim substring, then re-matched for exact offsets
`hybrid`	runs both; rules win on overlap

The LLM only locates, never rewrites, with a hallucination hard gate: a value that doesn't re-match the source is dropped (rather miss than be wrong); temperature=0 + response_format=json_object.

Chunking is block-aware and adapts to each document's heading structure: text.chunking.title_level_limit (auto by default, or a fixed level, 0 = top) picks which title level opens a new LLM call, so major chapters split calls while minor headings accumulate; tables always stand alone and every chunk is an exact slice of the document text. text.chunking.chunk_size sets the target chars per call.

📋 Default sensitive fields (`config/fields.yaml`)

field_type	label	default mask
`phone`	phone number `1[3-9]xxxxxxxxx`	`138****5678` (keep head 3 tail 4)
`id_card`	national ID number, GB 11643 checksum (toggleable)	`110105********1239` (keep head 6 tail 4)
`credit_code`	unified social credit code, GB 32100 checksum	`************000N` (keep tail 4)
`email`	email address	`**@*`
`stock_code`	A-share code starting `60/68/00/30`	`******`
`bank_account`	16-19 digit number (low confidence, off by default)	`6222***********5678` (keep head 4 tail 4)
`person_name`	surname dictionary + context heuristics	`***`
`company_name`	company-suffix dictionary + boundary pruning	`****`

Fields are fully configurable — add/remove/edit a line in config/fields.yaml; --fields a,b detects only what you name.

🖼️ Image masking

Images aren't only faces — they can hold company names, seals, screenshots of text. There are two complementary ways to decide what gets mosaiced; with no targets at all, no image is touched (rather miss than wrongly mosaic).

Class-driven (`--image-classes`)

--image-classes lists the objects to mask — anything a detection model can name (faces, people, door plates, storefront signs …):

class	description	dependency
`face`	faces: `auto` (default, YuNet auto-downloaded ~340KB on first use, offline fallback Haar) / `yunet` / `haar` / `yolo`	opencv (bundled)
anything else (`person`, `car`, …)	YOLO general object detection, filtered by class name; non-standard targets (door plates, signage) need custom weights	`--extra image-yolo` + `--image-model`

uv run pysanitize contract.pdf --mask-images --image-classes face
uv run pysanitize contract.pdf --mask-images --image-classes person,car --image-model yolov8n.pt

Text-driven (`--image-text`)

A company name or registered address rarely fits a class — it's text inside a logo or seal. Text in images is handled by OCR (--extra image-ocr), at either granularity:

uv run pysanitize contract.pdf --mask-images --image-text                        # ALL printed text
uv run pysanitize contract.pdf --mask-images --image-text company_name,address   # only matching fields

Bare --image-text mosaics every OCR'd text region (seals, screenshots, stamps)
With a field list, the same field detectors as the text pipeline run over the OCR'd text and only the matching spans are mosaiced
With neither form, field-driven masking follows the text field set (--fields); image.fields: [] in config/pipeline.yaml disables it
An explicit list may be a superset (an address stamped on a seal that never appears in the body text)
--image-classes is unaffected and still runs alongside
Near-full-page scan images are skipped (their text is already handled as document text)

The mosaic is a NEAREST block mosaic (default 16px) that covers only the detected boxes — everything outside is preserved.

📄 PDF redaction (`redacted.pdf`)

With --redact-pdf, the pipeline additionally writes redacted.pdf next to sanitized.md for PDF sources: the original layout is preserved while every detected span is truly deleted from the content stream and replaced with a mosaic (--redaction-style mosaic, default) or a solid box (block). Table borders and vector graphics stay; overlapping image pixels are cleared.

Coordinates come from MinerU's middle.json per-line bboxes; in-line hits are placed by proportional char width (CJK is effectively monospaced)
Tables: middle 3.x exposes no cell coordinates, so a hit in a table redacts the whole table bbox — conservative over-redaction, safe by design
Images: regions of images that were actually mosaiced are stamped back, so the PDF page matches images_masked/; unmasked images stay untouched
Scanned pages (no text layer) are naturally skipped by verification, which downgrades any leftover to a warning, never a failure
Opt-in: pass --redact-pdf, or set output.redact_pdf: true in config/pipeline.yaml to make it the default; office inputs never produce one

Why PyMuPDF (AGPL)

MinerU only reads PDFs — it has no writer, so re-rendering would lose fonts, table lines and backgrounds and still need a writer. PyMuPDF is the only dependency that reads and faithfully rewrites a PDF (true glyph deletion). It is AGPL-licensed: fine for internal tooling, but review the implications before embedding PySanitize in a closed-source product.

🔁 Recoverable masking (`--recoverable` / `--recover`)

By default masking is one-way. With --recoverable (needs uv sync --extra recover), the sanitized document looks exactly like a normal run — same user-configured placeholders (138****5678, ***, …) — but every value's ciphertext is recorded in audit.json, so the original can be restored later with the passphrase. Encryption is AES-256-GCM, keyed via scrypt from the passphrase; the same value always yields the same ciphertext within a run.

uv run pysanitize sample.pdf --recoverable                          # key auto-generated into .recover.key
uv run pysanitize sample.pdf --recoverable --recover-key s3cret     # ...also stored in .recover.key
PYSANITIZE_RECOVER_KEY=s3cret uv run pysanitize sample.pdf --recoverable   # ...also stored in .recover.key

# restore later — audit.json must sit beside the file (or pass --recover-audit);
# --recover just reads output/sample/.recover.key, no need to recall how the key was supplied:
uv run pysanitize output/sample/sanitized.md --recover
uv run pysanitize output/sample/redacted.pdf --recover
uv run pysanitize output/sample/redacted.pdf --recover --recover-key s3cret   # explicit override still works

However the passphrase is provided, the effective key is always written to output/<doc>/.recover.key (0600) at sanitize time — whether it came from --recover-key, $PYSANITIZE_RECOVER_KEY, or a fresh auto-generated draw — so .recover.key is the single source of truth and recovery only ever reads that file.

What the audit records (recovery mode only):

A recovery block — algorithm, KDF name, scrypt salt and parameters, ciphertext format. The salt is public by design; key material never touches the audit (it lives in .recover.key). Anyone holding the passphrase (or the keyfile) can recover; nobody without it can.
Per span: encrypted_value (the ENC(v1:…) ciphertext — masked_value stays the normal placeholder), start/end (offsets in the original text), md (where the placeholder sits in sanitized.md), and rects (the redacted PDF rectangles).

How restoration works:

Markdown restores exactly — the decrypted value is spliced back at the recorded md range (each splice verifies the placeholder is still there, so an edited document counts as unresolved instead of corrupting text).
PDF restores best-effort — the original glyphs were truly deleted at sanitize time; recovery clears the recorded rects and re-inserts the decrypted value with a fitted font: the values come back, the original typography does not. Images are not recoverable (mosaicing is destructive by nature).
Recovery is an independent consumer (pysanitize/recover/): it never imports the sanitize pipeline — only the audit + passphrase — and it never invents a key: with no passphrase available it fails instead.

⚠️ With --recoverable, audit.json carries the ciphertext of every sensitive value. Its secrecy reduces to passphrase strength: distribute it only where the document may be restored, and keep .recover.key (0600) as confidential as the data itself. The sanitized document itself remains as shareable as any normal run.

⚙️ LLM provider config

--model = filename of config/llm/<model>.yaml; --provider = a provider section in that yaml (openai: / claude: / openrouter: / …). One yaml can hold several sections — every provider is an OpenAI-compatible endpoint, so switching providers is just a flag change:

uv run pysanitize contract.pdf --detector hybrid --provider openrouter --model deepseek-v4-flash

api_key always uses a ${ENV_VAR} placeholder, expanded from the environment at runtime — plaintext keys never enter the repo
A missing section errors clearly: no 'openrouter' section in .../deepseek-v4-flash.yaml; have: openai
Global defaults live in config/pipeline.yaml under text.model / text.provider

🔒 Security boundaries & known limits

Scans (no text layer): the OCR'd Markdown is desensitized; redacted.pdf verification naturally finds nothing to remove there
Tables redact whole-table in the PDF (cell coordinates don't exist in middle 3.x); the sanitized.md output is still cell-precise
Person / company names are dictionary heuristics with limited precision; use hybrid for sensitive material
LLM hallucination: the verbatim re-match gate means misses are far likelier than false positives
Image masking is off by default: you must pass --image-classes and/or --image-text; bare --image-text mosaics all printed text in an image
No seal / handwriting / profile-face detection: text on seals rides OCR (lines under image.ocr.confidence are dropped); graphic signatures, logos and non-frontal faces have no dedicated detector — custom YOLO weights can add targets, but expect misses
No image-masking verification: unlike redacted.pdf (which is re-read for leftover values), masked images are not re-OCR'd to prove the text is gone — a detection miss stays invisible
Image OCR language defaults to ch (Chinese + Latin); other languages need image.ocr.lang in config/pipeline.yaml
Long-number false positives (18-digit figures in finance tables): checksums on by default + bank_account off by default
PyMuPDF is AGPL-licensed — used only for redacted.pdf; fine for internal tooling, review before closed-source distribution
--recoverable puts ciphertext in audit.json — the document stays shareable, but the audit + .recover.key (0600) must be guarded like the data itself
MinerU downloads models on first run; mineru[pipeline] is heavy (torch/opencv)

🛠️ Development

uv run pytest          # unit tests green

pysanitize/
├── parser/     MinerU wrapper (middle.json projection + v2 fallback) + ParsedDocument (line geometry / image pairing / offset mapping)
├── detector/   rules / llm / registry (overlap resolution) / image (face / YOLO classes / OCR text & field-driven)
├── masker/     text (offset masking) / image (mosaic)
├── redact/     offsets → page rects, PyMuPDF redaction + verification
├── recover/    reversible tokens (AES-GCM + scrypt) and pipeline-independent restoration
├── pipeline.py sanitize_document() orchestration (the only public entry)
├── cli.py      argparse CLI
├── llm/        LLM facade (openai / claude / openrouter providers)
└── report.py   audit.json / sensitive_report.json
config/         local overrides (git-ignored, optional): fields.yaml (field specs) / pipeline.yaml (all pipeline tunables) / llm/*.yaml (model config); built-in defaults apply without it

Extension points: add a field → edit fields.yaml; add a detector → write a class into the registry; add an image target → write a class into the build_detectors route; add an output format → add a renderer in M2. Every interface has a single method; nothing touches the core.

🗺️ Roadmap

M2: PDF redaction is shipped (redacted.pdf); remaining: DOCX/Excel in-place editing, metadata cleaning, anonymized output names
M3: WebUI (upload + task queue + progress)

Made with 🛡️ for safer documents · Report an issue

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
pysanitize		pysanitize
tests		tests
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
README_ZH.md		README_ZH.md
main.py		main.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Navigation Menu

Repository files navigation

🛡️ PySanitize

✨ Features

🔄 Pipeline

📦 Installation

🚀 Quick start

🖥️ Interactive TUI

🐍 Python API

🔍 Detection modes

📋 Default sensitive fields (`config/fields.yaml`)

🖼️ Image masking

Class-driven (`--image-classes`)

Text-driven (`--image-text`)

📄 PDF redaction (`redacted.pdf`)

Why PyMuPDF (AGPL)

🔁 Recoverable masking (`--recoverable` / `--recover`)

⚙️ LLM provider config

🔒 Security boundaries & known limits

🛠️ Development

🗺️ Roadmap

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🛡️ PySanitize

✨ Features

🔄 Pipeline

📦 Installation

🚀 Quick start

🖥️ Interactive TUI

🐍 Python API

🔍 Detection modes

📋 Default sensitive fields (config/fields.yaml)

🖼️ Image masking

Class-driven (--image-classes)

Text-driven (--image-text)

📄 PDF redaction (redacted.pdf)

Why PyMuPDF (AGPL)

🔁 Recoverable masking (--recoverable / --recover)

⚙️ LLM provider config

🔒 Security boundaries & known limits

🛠️ Development

🗺️ Roadmap

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

📋 Default sensitive fields (`config/fields.yaml`)

Class-driven (`--image-classes`)

Text-driven (`--image-text`)

📄 PDF redaction (`redacted.pdf`)

🔁 Recoverable masking (`--recoverable` / `--recover`)