diff --git a/docs/.mintignore b/docs/.mintignore new file mode 100644 index 0000000..ecc8053 --- /dev/null +++ b/docs/.mintignore @@ -0,0 +1,3 @@ +adr/ +privacy-capability-matrix.md +privacy-operations-roadmap.md diff --git a/docs/concepts/findings-and-ranges.mdx b/docs/concepts/findings-and-ranges.mdx new file mode 100644 index 0000000..18754fb --- /dev/null +++ b/docs/concepts/findings-and-ranges.mdx @@ -0,0 +1,75 @@ +--- +title: "Findings and text ranges" +description: "Understand finding metadata and the byte, code-point, and UTF-16 coordinate systems." +icon: "location-crosshairs" +--- + +`scan` returns an ordered list of findings. Each finding identifies what was +detected, the exact source text, its location, and the detector that produced +the result. + +## Finding fields + +| Field | Meaning | +| --- | --- | +| `entity_type` / `entityType` | Canonical entity name, such as `EMAIL` | +| `matched_text` / `matchedText` | Exact substring selected from the input | +| `byte_range` / `byteRange` | Range in UTF-8 bytes | +| `codepoint_range` / `codepointRange` | Range in Unicode code points | +| `utf16Range` | Range in UTF-16 code units; Node.js and browser/WASM only | +| `confidence` | Optional detector confidence in `0.0..=1.0` | +| `detector_name` / `detectorName` | Stable detector identifier | +| `detector_version` / `detectorVersion` | Optional detector implementation version | + +Rule-based built-in detectors currently omit confidence. + +## Range semantics + +Every range is: + +- zero-based; +- end-exclusive; +- explicitly named for its coordinate system; +- relative to the exact input text, without implicit Unicode normalization. + +For the text `👋 jane@example.com`, the email begins after an emoji and a space. +The same span has different offsets in each coordinate system: + +| Coordinate system | Email range | +| --- | --- | +| UTF-8 bytes | `5..21` | +| Unicode code points | `2..18` | +| UTF-16 code units | `3..19` | + +JavaScript strings use UTF-16 indexing, so Node.js and browser/WASM expose a +range that works directly with `String.prototype.slice`: + +```javascript +const text = "👋 jane@example.com"; +const finding = scan(text)[0]; + +const selected = text.slice( + finding.utf16Range.start, + finding.utf16Range.end, +); + +console.assert(selected === finding.matchedText); +``` + +## Supplied findings + +`transform` accepts caller-supplied findings but validates them before changing +text. The entire request fails when a finding is empty, reversed, out of bounds, +misaligned with a character boundary, inconsistent across coordinate systems, +or does not select its declared `matched_text`. + +JavaScript callers do not need to provide the derived `utf16Range` field when +supplying a finding to `transform`. + +## Duplicates and overlaps + +Exact duplicate findings collapse into one result before transformation. +Overlapping findings are resolved deterministically using structural span, +length, confidence when both values are present, source position, entity type, +and detector provenance. Selected transformations are returned in source +document order. diff --git a/docs/concepts/privacy-transformations.mdx b/docs/concepts/privacy-transformations.mdx new file mode 100644 index 0000000..d2a5ed5 --- /dev/null +++ b/docs/concepts/privacy-transformations.mdx @@ -0,0 +1,79 @@ +--- +title: "Privacy transformations" +description: "Choose between redaction, masking, removal, pseudonymization, and tokenization." +icon: "wand-magic-sparkles" +--- + +DataFog Core exposes concrete transformation strategies rather than claiming +that every output is anonymous. + +| Strategy | Output | Reversible | Provider required | Browser/WASM | +| --- | --- | --- | --- | --- | +| `redact` | `[ENTITY_TYPE]` | No | No | Supported | +| `mask` | Configured mask characters | No | No | Supported | +| `remove` | Empty replacement | No | No | Supported | +| `pseudonymize` | Keyed HMAC-SHA-256 pseudonym | No | Key provider | Unsupported | +| `tokenize` | Opaque `DFTOKENv1(...)` envelope | Yes | Token provider | Unsupported | + +## Redact + +Redaction replaces each selected finding with an unnumbered entity placeholder. + +```json +{ "default": { "strategy": "redact" } } +``` + +`jane@example.com` becomes `[EMAIL]`. + +## Mask + +Masking replaces every non-revealed Unicode code point. The default character +is `*`. A custom character must be exactly one non-whitespace, non-control +Unicode code point. + +```json +{ + "default": { + "strategy": "mask", + "character": "•", + "reveal": { "direction": "last", "count": 4 } + } +} +``` + +## Remove + +Removal deletes only the exact finding span. It does not normalize adjacent +whitespace. + +```json +{ "default": { "strategy": "remove" } } +``` + +## Pseudonymize + +Pseudonymization computes deterministic HMAC-SHA-256 over the exact UTF-8 +matched value using a provider-resolved 32-byte key. Use it when stable equality +under an intentionally scoped secret key is required. + +See [Pseudonymization](/guides/pseudonymization). + +## Tokenize + +Tokenization asks an application-owned provider to issue opaque token payloads. +The resulting envelope can later be restored under the same exact request +scope. + +See [Tokenization and restoration](/guides/tokenization-and-restoration). + +## Transformation records + +Each applied replacement produces a record with: + +- source ranges and detector provenance; +- the strategy and exact replacement; +- output ranges that select the replacement; +- resolved key metadata for pseudonymization; or +- resolved token metadata for tokenization. + +The record does not include `matched_text` or a plaintext-to-token mapping. diff --git a/docs/development.mdx b/docs/development.mdx new file mode 100644 index 0000000..92b8a34 --- /dev/null +++ b/docs/development.mdx @@ -0,0 +1,72 @@ +--- +title: "Development" +description: "Build DataFog Core and run the Rust and installed-package verification suites." +icon: "code" +--- + +## Clone the repository + +```bash +git clone https://github.com/DataFog/datafog-core.git +cd datafog-core +``` + +## Rust quality gates + +```bash +cargo fmt --all --check +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test --workspace --all-features +``` + +## Node.js installed-package test + +```bash +npm ci --prefix bindings/node +npm run test:package --prefix bindings/node +``` + +## Browser/WASM installed-package test + +```bash +rustup target add wasm32-unknown-unknown +cargo install wasm-bindgen-cli --version 0.2.127 --locked +npm ci --prefix bindings/wasm +npx --prefix bindings/wasm playwright install chromium +npm run test:package --prefix bindings/wasm +``` + +## Python installed-package test + +```bash +python -m venv .venv +.venv/bin/python -m pip install maturin +.venv/bin/maturin build --manifest-path bindings/python/Cargo.toml --release +.venv/bin/python -m pip install --force-reinstall target/wheels/*.whl +.venv/bin/python bindings/python/tests/test_installed.py +``` + +## Documentation preview + +The Mintlify content root is `docs/`. + +```bash +npm install --global mint +cd docs +mint validate +mint broken-links --check-anchors +mint dev --no-open +``` + +`mint dev --no-open` starts a local preview without opening a browser. + +## Repository structure + +```text +crates/core/ Rust scanning and transformation engine +bindings/python/ Python extension +bindings/node/ Node.js native binding +bindings/wasm/ Browser WebAssembly binding +fixtures/ Shared conformance fixtures +docs/ Architecture records and Mintlify documentation +``` diff --git a/docs/docs.json b/docs/docs.json new file mode 100644 index 0000000..7a58e1e --- /dev/null +++ b/docs/docs.json @@ -0,0 +1,61 @@ +{ + "$schema": "https://mintlify.com/docs.json", + "theme": "mint", + "name": "DataFog Core", + "description": "PII detection and privacy transformations for Rust, Python, Node.js, and browsers", + "colors": { + "primary": "#0F766E", + "light": "#0F766E", + "dark": "#2DD4BF" + }, + "navigation": { + "groups": [ + { + "group": "Get started", + "pages": [ + "index", + "get-started/installation", + "get-started/quickstart" + ] + }, + { + "group": "Core concepts", + "pages": [ + "concepts/findings-and-ranges", + "concepts/privacy-transformations" + ] + }, + { + "group": "Guides", + "pages": [ + "guides/configuration", + "guides/pseudonymization", + "guides/tokenization-and-restoration" + ] + }, + { + "group": "SDK reference", + "pages": [ + "reference/rust", + "reference/python", + "reference/node", + "reference/browser-wasm", + "reference/errors" + ] + }, + { + "group": "Contributing", + "pages": ["development"] + } + ], + "global": { + "anchors": [ + { + "anchor": "GitHub", + "href": "https://github.com/DataFog/datafog-core", + "icon": "github" + } + ] + } + } +} diff --git a/docs/get-started/installation.mdx b/docs/get-started/installation.mdx new file mode 100644 index 0000000..636323a --- /dev/null +++ b/docs/get-started/installation.mdx @@ -0,0 +1,55 @@ +--- +title: "Installation" +description: "Install DataFog Core for Rust or Python and review Node.js and browser package availability." +icon: "download" +--- + +## Package availability + +| Runtime | Distribution | Import | Availability | +| --- | --- | --- | --- | +| Rust | `datafog-core` | `datafog_core` | Published on crates.io | +| Python | `datafog-core` | `datafog_core` | Published on PyPI | +| Node.js | `@datafog/node` | `@datafog/node` | npm release pending | +| Browser/WASM | `@datafog/wasm` | `@datafog/wasm` | npm release pending | + + + +```bash Rust +cargo add datafog-core +``` + +```bash Python +python -m pip install datafog-core +``` + + + + + Do not use `pip install datafog` when following these pages. That command + installs the established DataFog Python library, not DataFog Core. + + +## Runtime requirements + +- Rust `1.88` or newer +- Python `3.10` or newer +- Node.js `24.x` for the native Node package +- A browser with WebAssembly support for the browser package + +## Work with unpublished JavaScript packages + +Until the npm releases are available, build and test the Node.js and browser +packages from the repository instead of adding them to a production project. + +```bash +git clone https://github.com/DataFog/datafog-core.git +cd datafog-core +``` + +See [Development](/development) for the package build and installed-package test +commands. + +## Next step + +Continue to the [Quickstart](/get-started/quickstart) to scan and redact text. diff --git a/docs/get-started/quickstart.mdx b/docs/get-started/quickstart.mdx new file mode 100644 index 0000000..5d44780 --- /dev/null +++ b/docs/get-started/quickstart.mdx @@ -0,0 +1,101 @@ +--- +title: "Quickstart" +description: "Scan text for PII and redact the detected value." +icon: "play" +--- + +This example detects an email address and replaces it with `[EMAIL]`. + + + +```rust Rust +use datafog_core::{ + scan, scan_and_transform, ScanAndTransformConfig, + TransformationConfig, TransformationStrategy, +}; + +let text = "Email jane@example.com"; +let findings = scan(text); + +assert_eq!(findings[0].entity_type, "EMAIL"); +assert_eq!(findings[0].matched_text, "jane@example.com"); + +let config = ScanAndTransformConfig::new(TransformationConfig::new( + TransformationStrategy::Redact, +)); +let result = scan_and_transform(text, &config).unwrap(); + +assert_eq!(result.text, "Email [EMAIL]"); +``` + +```python Python +from datafog_core import scan, scan_and_transform + +text = "Email jane@example.com" +findings = scan(text) + +assert findings[0].entity_type == "EMAIL" +assert findings[0].matched_text == "jane@example.com" + +result = scan_and_transform( + text, + {"transform": {"default": {"strategy": "redact"}}}, +) + +assert result.text == "Email [EMAIL]" +``` + +```javascript Node.js +import { scan, scanAndTransform } from "@datafog/node"; + +const text = "Email jane@example.com"; +const findings = scan(text); + +console.assert(findings[0].entityType === "EMAIL"); +console.assert(findings[0].matchedText === "jane@example.com"); + +const result = scanAndTransform(text, { + transform: { default: { strategy: "redact" } }, +}); + +console.assert(result.text === "Email [EMAIL]"); +``` + +```javascript Browser/WASM +import { init, scan, scanAndTransform } from "@datafog/wasm"; + +await init(); + +const text = "Email jane@example.com"; +const findings = scan(text); +const result = scanAndTransform(text, { + transform: { default: { strategy: "redact" } }, +}); + +console.assert(findings[0].entityType === "EMAIL"); +console.assert(result.text === "Email [EMAIL]"); +``` + + + + + The Node.js and browser/WASM examples target packages whose first npm release + is still pending. Rust and Python packages are currently published. + + +## Inspect the result + +A transformation result contains: + +- `text`: the transformed text; +- `transformations`: one ordered record for each applied replacement. + +Transformation records contain ranges, detector provenance, the selected +strategy, and the exact replacement. They intentionally do not repeat the +original matched PII. + +## Next steps + +- Learn how [findings and text ranges](/concepts/findings-and-ranges) work. +- Compare the available [privacy transformations](/concepts/privacy-transformations). +- Configure [entity selection, overrides, and allowlists](/guides/configuration). diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx new file mode 100644 index 0000000..6e67eb1 --- /dev/null +++ b/docs/guides/configuration.mdx @@ -0,0 +1,111 @@ +--- +title: "Configure transformations" +description: "Select entities, override strategies, add allowlists, and keep scan settings separate." +icon: "sliders" +--- + +Every transformation configuration requires a `default` strategy. + +```json +{ + "default": { "strategy": "redact" } +} +``` + +`scan_and_transform` wraps this configuration under `transform` and keeps +detection settings under `scan`: + +```json +{ + "scan": { "locale": "en-US" }, + "transform": { + "default": { "strategy": "redact" } + } +} +``` + +## Select entity types + +Omitting `entities` selects all supplied findings. A non-empty list selects only +exact, case-sensitive entity names. + +```json +{ + "default": { "strategy": "redact" }, + "entities": ["EMAIL", "PHONE"] +} +``` + +An empty `entities` list is invalid. + +## Override a strategy + +Overrides are exact and case-sensitive. Findings without an override use the +default strategy. + +```json +{ + "default": { "strategy": "redact" }, + "overrides": { + "PHONE": { + "strategy": "mask", + "reveal": { "direction": "last", "count": 4 } + } + } +} +``` + +## Exempt exact values + +Exact allowlists are scoped to an entity type and compare exact Unicode values +without normalization. + +```json +{ + "default": { "strategy": "redact" }, + "allow": { + "exact": { + "EMAIL": ["support@example.com"] + } + } +} +``` + +## Exempt values with regular expressions + +Regex allowlist patterns must match the full finding value. Matching is +case-sensitive by default. + +```json +{ + "default": { "strategy": "redact" }, + "allow": { + "regex": { + "EMAIL": [ + { "pattern": ".+@example\\.org" }, + { "pattern": ".+@example\\.com", "case_sensitive": false } + ] + } + } +} +``` + +## Evaluation order + +DataFog Core processes a request in this order: + +1. Validate the complete configuration. +2. Select entity types. +3. Apply exact and regex allowlists. +4. Resolve duplicates and overlaps. +5. Choose the per-entity override or default strategy. +6. Transform selected findings in document order. + +Valid configuration for an unselected entity remains dormant. Invalid fields +are rejected even when their entity is not selected. + +## Strict input + +Unknown fields and explicit `null` values are rejected. Empty structural maps +are treated as omission, but empty semantic values such as an empty entity +selection, key reference, or token reference are invalid. diff --git a/docs/guides/pseudonymization.mdx b/docs/guides/pseudonymization.mdx new file mode 100644 index 0000000..1f6859a --- /dev/null +++ b/docs/guides/pseudonymization.mdx @@ -0,0 +1,105 @@ +--- +title: "Pseudonymization" +description: "Create deterministic one-way pseudonyms with application-managed keys." +icon: "key" +--- + +Pseudonymization replaces a finding with a deterministic keyed digest. DataFog +Core uses HMAC-SHA-256 over the exact UTF-8 matched value and returns the full +digest as standard padded Base64. + + + Pseudonymization is not encryption and is not reversible. Its privacy and + linkage boundary depend on how your application scopes and protects keys. + + +## Configure the strategy + +```json +{ + "default": { + "strategy": "pseudonymize", + "key_ref": "customers/email", + "key_version": "7" + } +} +``` + +- `key_ref` is required and selects provider-owned key material. +- `key_version` is optional and requests a provider version. +- The provider must return exactly 32 key bytes and a non-empty concrete + resolved version. + +The key reference and resolved version appear in transformation records. Key +material never appears in serialized configuration, results, errors, or debug +output. + +## Python provider + +```python +import asyncio + +from datafog_core import PrivacyManager + + +class KeyProvider: + async def resolve_key(self, key_ref, key_version): + key = await load_key_from_your_kms(key_ref, key_version) + return {"key": key, "resolved_version": "7"} + + +async def main(): + manager = PrivacyManager(KeyProvider()) + return await manager.scan_and_transform( + "Email jane@example.com", + { + "transform": { + "default": { + "strategy": "pseudonymize", + "key_ref": "customers/email", + } + } + }, + ) + + +result = asyncio.run(main()) +``` + +## Node.js provider + +```javascript +import { PrivacyManager } from "@datafog/node"; + +const manager = new PrivacyManager({ + async resolveKey({ keyRef, keyVersion }) { + return { + key: await loadKeyFromYourKms(keyRef, keyVersion), + resolvedVersion: "7", + }; + }, +}); + +const result = await manager.scanAndTransform("Email jane@example.com", { + transform: { + default: { + strategy: "pseudonymize", + key_ref: "customers/email", + }, + }, +}); +``` + +The example `loadKeyFromYourKms` functions are application code. DataFog Core +does not ship cloud-specific key-provider adapters. + +## Request behavior + +- Every distinct selected key reference/version is resolved once per request. +- All keys resolve and validate before text is changed. +- Provider failures return no partial transformation result. +- The same exact value and key produce the same pseudonym. +- Changing the value or key changes the pseudonym. + +Browser/WASM deliberately returns `unsupported_strategy` for pseudonymization +because it has no accepted host-managed key-custody boundary. diff --git a/docs/guides/tokenization-and-restoration.mdx b/docs/guides/tokenization-and-restoration.mdx new file mode 100644 index 0000000..c0576fa --- /dev/null +++ b/docs/guides/tokenization-and-restoration.mdx @@ -0,0 +1,117 @@ +--- +title: "Tokenization and restoration" +description: "Issue opaque reversible tokens through an application-owned provider and restore them atomically." +icon: "rotate" +--- + +Tokenization is the reversible privacy operation. DataFog Core defines request +validation, provider batching, a canonical token envelope, and atomic text +mutation. Your provider owns storage or reversible cryptography, authorization, +lifecycle, retries, and audit logging. + +## Configure tokenization + +```json +{ + "default": { + "strategy": "tokenize", + "token_ref": "customers/default" + } +} +``` + +Selected tokenization and every restoration request require an exact, +case-sensitive request scope: + +```json +{ "scope": "tenant-a" } +``` + +## Provider contract + +A token provider implements two asynchronous batch methods: + +- `tokenize_batch(scope, items)` / `tokenizeBatch(scope, items)` +- `restore_batch(scope, items)` / `restoreBatch(scope, items)` + +Each tokenization item contains an opaque request `id`, the `exact_value`, and +the configured `token_ref`. Return the same `id`, opaque payload bytes, and a +concrete resolved profile version. + +Each restoration item contains an `id`, `token_ref`, resolved version, and +opaque payload. Return the same `id` and restored value. + +## Python round trip + +```python +import asyncio + +from datafog_core import PrivacyManager + + +async def round_trip(token_provider): + manager = PrivacyManager(None, token_provider=token_provider) + context = {"scope": "tenant-a"} + + tokenized = await manager.scan_and_transform( + "Email jane@example.com", + { + "transform": { + "default": { + "strategy": "tokenize", + "token_ref": "customers/default", + } + } + }, + context, + ) + + return await manager.restore(tokenized.text, context) + + +restored = asyncio.run(round_trip(token_provider)) +assert restored.text == "Email jane@example.com" +``` + +## Node.js round trip + +```javascript +import { PrivacyManager } from "@datafog/node"; + +const manager = new PrivacyManager({ tokenProvider }); +const context = { scope: "tenant-a" }; + +const tokenized = await manager.scanAndTransform( + "Email jane@example.com", + { + transform: { + default: { + strategy: "tokenize", + token_ref: "customers/default", + }, + }, + }, + context, +); + +const restored = await manager.restore(tokenized.text, context); +console.assert(restored.text === "Email jane@example.com"); +``` + +## Atomic and non-recursive behavior + +- Repeated source values are separate tokenization items and may receive + different tokens. +- Identical envelopes are deduplicated before restoration provider calls. +- Every canonical token in the supplied text is restored, or no result is + returned. +- Nested tokenization and recursive restoration are rejected. +- Partial restoration and ignore-failure modes are not available. + +Token envelopes use the canonical +`DFTOKENv1():..` form with unpadded +Base64URL components. Treat the envelope as opaque application data; do not +parse or construct it yourself. + +Browser/WASM rejects selected tokenization and every restoration call with +`unsupported_strategy`. diff --git a/docs/index.mdx b/docs/index.mdx new file mode 100644 index 0000000..0a4f8a1 --- /dev/null +++ b/docs/index.mdx @@ -0,0 +1,69 @@ +--- +title: "DataFog Core" +description: "Detect PII and apply explicit privacy transformations from Rust, Python, Node.js, and browsers." +icon: "shield-halved" +--- + +DataFog Core is a multi-runtime engine for detecting sensitive text and applying +privacy transformations. The implementation lives in Rust, with thin bindings +for Python, Node.js, and browser WebAssembly. + + + DataFog Core is distributed as `datafog-core`. It is separate from the + established `datafog` Python package and does not reproduce its legacy API. + + +## What it does + + + + Scan text for structured findings with explicit ranges and detector provenance. + + + Redact, mask, remove, pseudonymize, or tokenize selected findings. + + + Select entity types, apply per-entity overrides, and exempt approved values. + + + Restore provider-issued tokens atomically under an exact authorization scope. + + + +## Supported entities + +The built-in detector recognizes: + +- `EMAIL` +- `PHONE` +- `SSN` +- `CREDIT_CARD` +- `IP_ADDRESS` +- `DATE` +- `ZIP_CODE` + +Built-in entity names are uppercase. The finding contract remains extensible so +custom detectors can introduce additional entity names in the future. + +## Operation model + +```text +scan(text) -> findings +transform(text, findings, …) -> transformed text + records +scan_and_transform(text, …) -> scan, then transform +restore(text, context) -> restored text + records +``` + +`transform` never scans implicitly. Use `scan_and_transform` when you want the +explicit scan-then-transform convenience operation. + +## Start here + + + + Choose the distribution for your runtime. + + + Detect and redact an email address in a few lines. + + diff --git a/docs/reference/browser-wasm.mdx b/docs/reference/browser-wasm.mdx new file mode 100644 index 0000000..ac8163a --- /dev/null +++ b/docs/reference/browser-wasm.mdx @@ -0,0 +1,55 @@ +--- +title: "Browser / WASM" +description: "Browser WebAssembly initialization, stateless operations, and provider-backed limitations." +icon: "globe" +--- + + + The first `@datafog/wasm` npm release is pending. The API documented here is + implemented and tested in a real browser from the repository. + + +## Initialize the module + +Call `init` once before using another operation. + +```javascript +import { init, scan, scanAndTransform, transform } from "@datafog/wasm"; + +await init(); +``` + +## Operations + +```typescript +scan(text: string, config?: ScanConfig): Finding[] +transform( + text: string, + findings: FindingInput[], + config: TransformationConfig, +): TransformResult +scanAndTransform(text: string, config: ScanAndTransformConfig): TransformResult +``` + +The browser binding supports the stateless `redact`, `mask`, and `remove` +strategies. + +## JavaScript-native ranges + +Findings contain `utf16Range`. Transformation records contain +`sourceUtf16Range` and `outputUtf16Range`. These zero-based, end-exclusive +ranges work directly with `String.prototype.slice`. + +## Unsupported provider operations + +Browser/WASM does not accept key or token providers. It returns +`unsupported_strategy` when pseudonymization or tokenization is selected and +for every restoration call. + +This is an intentional security boundary, not a missing fallback. Provider +credentials and key custody require a separately designed host integration. + +## Errors + +The JavaScript wrapper throws `DataFogError` with the same stable fields as the +Node.js binding. See [Errors](/reference/errors). diff --git a/docs/reference/errors.mdx b/docs/reference/errors.mdx new file mode 100644 index 0000000..72bfba7 --- /dev/null +++ b/docs/reference/errors.mdx @@ -0,0 +1,81 @@ +--- +title: "Errors" +description: "Handle stable error codes, machine-readable reasons, request paths, and provider failures." +icon: "triangle-exclamation" +--- + +Transformation and restoration failures are atomic: DataFog Core returns no +partially transformed result. + +## Error fields + +| Field | Meaning | +| --- | --- | +| `code` | Stable top-level error category | +| `reason` | Optional machine-readable detail | +| `message` | Sanitized human-readable explanation | +| `path` | Optional RFC 6901 request path | +| `finding_index` / `findingIndex` | Optional index of an invalid supplied finding | + +Error messages and metadata do not include matched PII, key material, +credentials, token payloads, or restored plaintext. + +## Error codes + +### Request validation + +- `invalid_configuration` +- `invalid_finding` + +### Key providers + +- `key_provider_required` +- `key_not_found` +- `key_access_denied` +- `key_provider_unavailable` +- `invalid_key_material` +- `key_provider_error` + +### Token providers and envelopes + +- `token_provider_required` +- `invalid_token` +- `unsupported_token_version` +- `token_not_found` +- `token_expired` +- `token_access_denied` +- `invalid_token_material` +- `token_provider_unavailable` +- `token_provider_error` + +### Capability and engine failures + +- `unsupported_strategy` +- `internal_error` + +## JavaScript example + +```javascript +try { + scanAndTransform(text, config); +} catch (error) { + if (error instanceof DataFogError) { + console.error(error.code, error.path); + } +} +``` + +## Python example + +```python +from datafog_core import DataFogConfigurationError + +try: + scan_and_transform(text, config) +except DataFogConfigurationError as error: + print(error.code, error.reason, error.path) +``` + + + Branch on `code` and `reason`, not on the human-readable message. + diff --git a/docs/reference/node.mdx b/docs/reference/node.mdx new file mode 100644 index 0000000..a258343 --- /dev/null +++ b/docs/reference/node.mdx @@ -0,0 +1,75 @@ +--- +title: "Node.js" +description: "Node.js functions, TypeScript types, JavaScript ranges, and the asynchronous PrivacyManager." +icon: "node-js" +--- + + + The first `@datafog/node` npm release is pending. The API documented here is + implemented and tested in the repository. + + +```javascript +import { + DataFogError, + PrivacyManager, + scan, + scanAndTransform, + transform, +} from "@datafog/node"; +``` + +## Synchronous functions + +```typescript +scan(text: string, config?: ScanConfig): Finding[] +transform( + text: string, + findings: FindingInput[], + config: TransformationConfig, +): TransformResult +scanAndTransform(text: string, config: ScanAndTransformConfig): TransformResult +``` + +## Findings and ranges + +Node.js uses camelCase fields. Findings include `byteRange`, `codepointRange`, +and `utf16Range`. Use the UTF-16 range with native JavaScript string slicing. + +```javascript +const finding = scan("👋 jane@example.com")[0]; +console.assert(finding.utf16Range.start === 3); +``` + +Caller-supplied `FindingInput` objects do not require `utf16Range`. + +## Asynchronous manager + +Use `PrivacyManager` for provider-backed pseudonymization, tokenization, and +restoration: + +```typescript +new PrivacyManager( + provider: KeyProvider | PrivacyManagerProviders, + tokenProvider?: TokenProvider, +) +``` + +```typescript +await manager.transform(text, findings, config, context?) +await manager.scanAndTransform(text, config, context?) +await manager.restore(text, context) +``` + +## Native distribution targets + +The package build is configured for: + +- macOS x64 and Apple Silicon; +- Linux x64 and ARM64; +- Windows x64. + +## Errors + +JavaScript operations throw `DataFogError` with stable `code`, optional +`reason`, optional RFC 6901 `path`, and optional `findingIndex` properties. diff --git a/docs/reference/python.mdx b/docs/reference/python.mdx new file mode 100644 index 0000000..a43a288 --- /dev/null +++ b/docs/reference/python.mdx @@ -0,0 +1,86 @@ +--- +title: "Python" +description: "Python functions, result objects, exceptions, and the asynchronous PrivacyManager." +icon: "python" +--- + +```bash +python -m pip install datafog-core +``` + +```python +import datafog_core +``` + +## Synchronous functions + +```python +scan(text: str, config: dict | None = None) -> list[Finding] +transform(text: str, findings: list[Finding], config: dict) -> TransformResult +scan_and_transform(text: str, config: dict) -> TransformResult +``` + +`transform` uses explicit findings. `scan_and_transform` accepts the divided +configuration envelope with `scan` and `transform` sections. + +## Result objects + +Python exposes immutable objects with snake_case attributes: + +- `TextRange(start, end)` +- `Finding` +- `Transformation` +- `TransformResult` +- `Restoration` +- `RestoreResult` + +```python +finding.entity_type +finding.matched_text +finding.byte_range.start +finding.codepoint_range.end + +result.text +result.transformations +``` + +## Asynchronous manager + +```python +PrivacyManager(provider=None, token_provider=None) +``` + +The manager exposes awaitable methods: + +```python +await manager.transform(text, findings, config, context=None) +await manager.scan_and_transform(text, config, context=None) +await manager.restore(text, context) +``` + +The key provider must implement: + +```python +async def resolve_key(key_ref: str, key_version: str | None) -> dict: + ... +``` + +The token provider must implement: + +```python +async def tokenize_batch(scope: str, items: list[dict]) -> list[dict]: + ... + +async def restore_batch(scope: str, items: list[dict]) -> list[dict]: + ... +``` + +## Exceptions + +- `DataFogConfigurationError` +- `DataFogFindingError` +- `DataFogKeyProviderError` +- `DataFogInternalError` + +Exceptions expose stable `code`, optional `reason`, optional `path`, and +optional `finding_index` attributes. See [Errors](/reference/errors). diff --git a/docs/reference/rust.mdx b/docs/reference/rust.mdx new file mode 100644 index 0000000..ced88e0 --- /dev/null +++ b/docs/reference/rust.mdx @@ -0,0 +1,95 @@ +--- +title: "Rust" +description: "Rust functions, configuration types, results, and provider-backed operations." +icon: "gear" +--- + +```bash +cargo add datafog-core +``` + +```rust +use datafog_core::*; +``` + +## Stateless operations + +### `scan` + +```rust +pub fn scan(text: &str) -> Vec +``` + +Scan text with the default scan configuration. + +### `scan_with_config` + +```rust +pub fn scan_with_config(text: &str, config: &ScanConfig) -> Vec +``` + +### `transform` + +```rust +pub fn transform( + text: &str, + findings: &[Finding], + config: &TransformationConfig, +) -> Result +``` + +Transform caller-supplied findings without scanning implicitly. Provider-backed +strategies return a provider-required error from this stateless function. + +### `scan_and_transform` + +```rust +pub fn scan_and_transform( + text: &str, + config: &ScanAndTransformConfig, +) -> Result +``` + +## Primary configuration types + +- `ScanConfig` +- `ScanAndTransformConfig` +- `TransformationConfig` +- `TransformationStrategy` +- `MaskConfig` and `MaskReveal` +- `PseudonymizeConfig` +- `TokenizeConfig` +- `PrivacyContext` + +Rust constructors validate semantic values and return typed errors where +configuration can be invalid. + +## Results + +`Finding` exposes entity metadata, matched text, byte and code-point ranges, +optional confidence, and detector provenance. + +`TransformResult` contains transformed `text` and ordered `transformations`. +`RestoreResult` contains restored `text` and ordered `restorations`. + +## Provider-backed manager + +`PrivacyManager` composes key and token provider capabilities. + +```rust +let manager = PrivacyManager::new(key_provider) + .with_token_provider(token_provider); +``` + +Use: + +- `transform` or `scan_and_transform` for key-backed pseudonymization; +- `transform_with_context` or `scan_and_transform_with_context` when + tokenization may be selected; +- `restore` for authorized token restoration. + +Implement the `KeyProvider` and `TokenProvider` traits in application code. +DataFog Core ships no cloud-, vault-, or database-specific provider. + +For exact public definitions, see the +[crate source](https://github.com/DataFog/datafog-core/blob/main/crates/core/src/lib.rs).