Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/.mintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
adr/
privacy-capability-matrix.md
privacy-operations-roadmap.md
75 changes: 75 additions & 0 deletions docs/concepts/findings-and-ranges.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: "Findings and text ranges"
description: "Understand finding metadata and the byte, code-point, and UTF-16 coordinate systems."
icon: "location-crosshairs"
---

`scan` returns an ordered list of findings. Each finding identifies what was
detected, the exact source text, its location, and the detector that produced
the result.

## Finding fields

| Field | Meaning |
| --- | --- |
| `entity_type` / `entityType` | Canonical entity name, such as `EMAIL` |
| `matched_text` / `matchedText` | Exact substring selected from the input |
| `byte_range` / `byteRange` | Range in UTF-8 bytes |
| `codepoint_range` / `codepointRange` | Range in Unicode code points |
| `utf16Range` | Range in UTF-16 code units; Node.js and browser/WASM only |
| `confidence` | Optional detector confidence in `0.0..=1.0` |
| `detector_name` / `detectorName` | Stable detector identifier |
| `detector_version` / `detectorVersion` | Optional detector implementation version |

Rule-based built-in detectors currently omit confidence.

## Range semantics

Every range is:

- zero-based;
- end-exclusive;
- explicitly named for its coordinate system;
- relative to the exact input text, without implicit Unicode normalization.

For the text `👋 jane@example.com`, the email begins after an emoji and a space.
The same span has different offsets in each coordinate system:

| Coordinate system | Email range |
| --- | --- |
| UTF-8 bytes | `5..21` |
| Unicode code points | `2..18` |
| UTF-16 code units | `3..19` |

JavaScript strings use UTF-16 indexing, so Node.js and browser/WASM expose a
range that works directly with `String.prototype.slice`:

```javascript
const text = "👋 jane@example.com";
const finding = scan(text)[0];

const selected = text.slice(
finding.utf16Range.start,
finding.utf16Range.end,
);

console.assert(selected === finding.matchedText);
```

## Supplied findings

`transform` accepts caller-supplied findings but validates them before changing
text. The entire request fails when a finding is empty, reversed, out of bounds,
misaligned with a character boundary, inconsistent across coordinate systems,
or does not select its declared `matched_text`.

JavaScript callers do not need to provide the derived `utf16Range` field when
supplying a finding to `transform`.

## Duplicates and overlaps

Exact duplicate findings collapse into one result before transformation.
Overlapping findings are resolved deterministically using structural span,
length, confidence when both values are present, source position, entity type,
and detector provenance. Selected transformations are returned in source
document order.
79 changes: 79 additions & 0 deletions docs/concepts/privacy-transformations.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
title: "Privacy transformations"
description: "Choose between redaction, masking, removal, pseudonymization, and tokenization."
icon: "wand-magic-sparkles"
---

DataFog Core exposes concrete transformation strategies rather than claiming
that every output is anonymous.

| Strategy | Output | Reversible | Provider required | Browser/WASM |
| --- | --- | --- | --- | --- |
| `redact` | `[ENTITY_TYPE]` | No | No | Supported |
| `mask` | Configured mask characters | No | No | Supported |
| `remove` | Empty replacement | No | No | Supported |
| `pseudonymize` | Keyed HMAC-SHA-256 pseudonym | No | Key provider | Unsupported |
| `tokenize` | Opaque `DFTOKENv1(...)` envelope | Yes | Token provider | Unsupported |

## Redact

Redaction replaces each selected finding with an unnumbered entity placeholder.

```json
{ "default": { "strategy": "redact" } }
```

`jane@example.com` becomes `[EMAIL]`.

## Mask

Masking replaces every non-revealed Unicode code point. The default character
is `*`. A custom character must be exactly one non-whitespace, non-control
Unicode code point.

```json
{
"default": {
"strategy": "mask",
"character": "•",
"reveal": { "direction": "last", "count": 4 }
}
}
```

## Remove

Removal deletes only the exact finding span. It does not normalize adjacent
whitespace.

```json
{ "default": { "strategy": "remove" } }
```

## Pseudonymize

Pseudonymization computes deterministic HMAC-SHA-256 over the exact UTF-8
matched value using a provider-resolved 32-byte key. Use it when stable equality
under an intentionally scoped secret key is required.

See [Pseudonymization](/guides/pseudonymization).

## Tokenize

Tokenization asks an application-owned provider to issue opaque token payloads.
The resulting envelope can later be restored under the same exact request
scope.

See [Tokenization and restoration](/guides/tokenization-and-restoration).

## Transformation records

Each applied replacement produces a record with:

- source ranges and detector provenance;
- the strategy and exact replacement;
- output ranges that select the replacement;
- resolved key metadata for pseudonymization; or
- resolved token metadata for tokenization.

The record does not include `matched_text` or a plaintext-to-token mapping.
72 changes: 72 additions & 0 deletions docs/development.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: "Development"
description: "Build DataFog Core and run the Rust and installed-package verification suites."
icon: "code"
---

## Clone the repository

```bash
git clone https://github.com/DataFog/datafog-core.git
cd datafog-core
```

## Rust quality gates

```bash
cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features
```

## Node.js installed-package test

```bash
npm ci --prefix bindings/node
npm run test:package --prefix bindings/node
```

## Browser/WASM installed-package test

```bash
rustup target add wasm32-unknown-unknown
cargo install wasm-bindgen-cli --version 0.2.127 --locked
npm ci --prefix bindings/wasm
npx --prefix bindings/wasm playwright install chromium
npm run test:package --prefix bindings/wasm
```

## Python installed-package test

```bash
python -m venv .venv
.venv/bin/python -m pip install maturin
.venv/bin/maturin build --manifest-path bindings/python/Cargo.toml --release
.venv/bin/python -m pip install --force-reinstall target/wheels/*.whl
.venv/bin/python bindings/python/tests/test_installed.py
```

## Documentation preview

The Mintlify content root is `docs/`.

```bash
npm install --global mint
cd docs
mint validate
mint broken-links --check-anchors
mint dev --no-open
```

`mint dev --no-open` starts a local preview without opening a browser.

## Repository structure

```text
crates/core/ Rust scanning and transformation engine
bindings/python/ Python extension
bindings/node/ Node.js native binding
bindings/wasm/ Browser WebAssembly binding
fixtures/ Shared conformance fixtures
docs/ Architecture records and Mintlify documentation
```
61 changes: 61 additions & 0 deletions docs/docs.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"$schema": "https://mintlify.com/docs.json",
"theme": "mint",
"name": "DataFog Core",
"description": "PII detection and privacy transformations for Rust, Python, Node.js, and browsers",
"colors": {
"primary": "#0F766E",
"light": "#0F766E",
"dark": "#2DD4BF"
},
"navigation": {
"groups": [
{
"group": "Get started",
"pages": [
"index",
"get-started/installation",
"get-started/quickstart"
]
},
{
"group": "Core concepts",
"pages": [
"concepts/findings-and-ranges",
"concepts/privacy-transformations"
]
},
{
"group": "Guides",
"pages": [
"guides/configuration",
"guides/pseudonymization",
"guides/tokenization-and-restoration"
]
},
{
"group": "SDK reference",
"pages": [
"reference/rust",
"reference/python",
"reference/node",
"reference/browser-wasm",
"reference/errors"
]
},
{
"group": "Contributing",
"pages": ["development"]
}
],
"global": {
"anchors": [
{
"anchor": "GitHub",
"href": "https://github.com/DataFog/datafog-core",
"icon": "github"
}
]
}
}
}
55 changes: 55 additions & 0 deletions docs/get-started/installation.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: "Installation"
description: "Install DataFog Core for Rust or Python and review Node.js and browser package availability."
icon: "download"
---

## Package availability

| Runtime | Distribution | Import | Availability |
| --- | --- | --- | --- |
| Rust | `datafog-core` | `datafog_core` | Published on crates.io |
| Python | `datafog-core` | `datafog_core` | Published on PyPI |
| Node.js | `@datafog/node` | `@datafog/node` | npm release pending |
| Browser/WASM | `@datafog/wasm` | `@datafog/wasm` | npm release pending |

<CodeGroup>

```bash Rust
cargo add datafog-core
```

```bash Python
python -m pip install datafog-core
```

</CodeGroup>

<Warning>
Do not use `pip install datafog` when following these pages. That command
installs the established DataFog Python library, not DataFog Core.
</Warning>

## Runtime requirements

- Rust `1.88` or newer
- Python `3.10` or newer
- Node.js `24.x` for the native Node package
- A browser with WebAssembly support for the browser package

## Work with unpublished JavaScript packages

Until the npm releases are available, build and test the Node.js and browser
packages from the repository instead of adding them to a production project.

```bash
git clone https://github.com/DataFog/datafog-core.git
cd datafog-core
```

See [Development](/development) for the package build and installed-package test
commands.

## Next step

Continue to the [Quickstart](/get-started/quickstart) to scan and redact text.
Loading
Loading