Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

44 changes: 39 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,18 @@ Both ranges use zero-based, end-exclusive offsets. The byte range addresses the
UTF-8 input; the code-point range addresses Unicode scalar values. Rule-based
detectors currently report no confidence score.

The initial transformation strategies are `redact`, `mask`, and `remove`.
The transformation strategies are `redact`, `mask`, `remove`, and
`pseudonymize` in Rust, Python, and Node.js.
Redaction uses an unnumbered `[ENTITY_TYPE]` placeholder, masking supports full
or leading/trailing reveal modes, and removal deletes only the exact finding
span. `transform` requires explicit findings; `scan_and_transform` (or
span. Pseudonymization uses provider-resolved 256-bit keys and deterministic
HMAC-SHA-256 tokens; it is deliberately unsupported in browser WASM.
`transform` requires explicit findings; `scan_and_transform` (or
`scanAndTransform` in JavaScript) is the explicit scan-then-transform
convenience. Results include the transformed text and an ordered record for
every applied replacement, including its output byte and code-point ranges.
every applied replacement, including its source metadata and output byte and
code-point ranges. Transformation records never include the original matched
text.

Transformation calls require an envelope with a default strategy. It can also
select entity types, override the strategy per entity, and exempt exact or
Expand Down Expand Up @@ -86,7 +91,9 @@ python -m pip install datafog-core
```

```python
from datafog_core import scan, scan_and_transform
import asyncio

from datafog_core import PrivacyManager, scan, scan_and_transform

findings = scan("Email jane@example.com")
print(findings[0].entity_type) # EMAIL
Expand All @@ -111,21 +118,48 @@ masked = scan_and_transform(
},
)
assert masked.text == "Email ************.com"

class KeyProvider:
async def resolve_key(self, key_ref, key_version):
return {"key": load_32_byte_key(key_ref, key_version), "resolved_version": "7"}

async def pseudonymize():
return await PrivacyManager(KeyProvider()).scan_and_transform(
"Email jane@example.com",
{
"transform": {
"default": {"strategy": "pseudonymize", "key_ref": "customers/email"}
}
},
)

pseudonymized = asyncio.run(pseudonymize())
```

### Node.js

`@datafog/node` will install as a native package once its npm release is published.

```js
import { scan, scanAndTransform } from "@datafog/node";
import { PrivacyManager, scan, scanAndTransform } from "@datafog/node";

console.log(scan("Email jane@example.com"));
console.log(
scanAndTransform("Email jane@example.com", {
transform: { default: { strategy: "redact" } },
}).text,
);

const manager = new PrivacyManager({
async resolveKey({ keyRef, keyVersion }) {
return { key: await load32ByteKey(keyRef, keyVersion), resolvedVersion: "7" };
},
});
const pseudonymized = await manager.scanAndTransform("Email jane@example.com", {
transform: {
default: { strategy: "pseudonymize", key_ref: "customers/email" },
},
});
```

The release includes prebuilt binaries for macOS (Intel and Apple Silicon), Linux (x64 and ARM64), and Windows x64.
Expand Down
45 changes: 44 additions & 1 deletion bindings/node/dts-header.d.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
/** Canonical built-in values are uppercase, but custom detectors may add values. */
export type EntityType = string;

export type TransformationStrategy = "redact" | "mask" | "remove";
export type TransformationStrategy =
| "redact"
| "mask"
| "remove"
| "pseudonymize";

export interface MaskRevealConfig {
readonly direction: "first" | "last";
Expand All @@ -11,6 +15,11 @@ export interface MaskRevealConfig {
export type TransformationStrategyConfig =
| { readonly strategy: "redact" }
| { readonly strategy: "remove" }
| {
readonly strategy: "pseudonymize";
readonly key_ref: string;
readonly key_version?: string;
}
| {
readonly strategy: "mask";
readonly character?: string;
Expand Down Expand Up @@ -46,6 +55,13 @@ export interface ScanAndTransformConfig {
export type DataFogErrorCode =
| "invalid_configuration"
| "invalid_finding"
| "key_provider_required"
| "key_not_found"
| "key_access_denied"
| "key_provider_unavailable"
| "invalid_key_material"
| "key_provider_error"
| "unsupported_strategy"
| "internal_error";

export declare class DataFogError extends Error {
Expand All @@ -54,3 +70,30 @@ export declare class DataFogError extends Error {
readonly path?: string;
readonly findingIndex?: number;
}

export interface KeyProviderRequest {
readonly keyRef: string;
readonly keyVersion?: string;
}

export interface KeyProviderResponse {
readonly key: Uint8Array;
readonly resolvedVersion: string;
}

export interface KeyProvider {
resolveKey(request: KeyProviderRequest): Promise<KeyProviderResponse>;
}

export declare class PrivacyManager {
constructor(provider: KeyProvider);
transform(
text: string,
findings: Finding[],
config: TransformationConfig,
): Promise<TransformResult>;
scanAndTransform(
text: string,
config: ScanAndTransformConfig,
): Promise<TransformResult>;
}
Loading
Loading