English | 简体中文

pgdocket

Durable, versioned, calibrated AI decisions for Postgres. Declare a semantic decision once ("is this complaint about debt collection?"). pgdocket keeps the answer stored as data, versioned, calibrated against your own labels, and honest about its uncertainty, in any Postgres and with any decision model.

Status: Phase 1 in progress. Working: pgdocket check measures how trustworthy an AI decision is on your table; the ledger stores decisions in Postgres and keeps them current as rows change; pgdocket estimate gives counts with confidence intervals. Not built yet: version comparison, other backends' adapters, a published release. The repository also contains the design (REPORT.md, ADRs) and the Phase 0 benchmark (findings). Progress is tracked in issues and milestones.

The problem

pg-jev lets you filter rows in plain language, such as WHERE jev(tickets, 'the customer threatens to cancel'), using TypeSafe's Jev decision model. That's good for exploring. To rely on the answers in production, they need to be stable, comparable and trustworthy, and today they aren't:

• they disappear when the database session ends, so every session judges the whole table again;
• they change when someone adds a label option, or switches model or backend;
• they change when you simply ask again, or send the row alongside different rows;
• a score of 0.7 doesn't mean 70 %, and count(*) over AI labels has no error bar;
• they don't run on managed hosts like Supabase, Neon or RDS.

Model calls now cost roughly $6–25 per million rows (depending on text length and rows per request), so the bottleneck is no longer cost. It's whether people can trust and audit the answers.

What Phase 0 measured

2,000 real consumer-complaint narratives, judged by jev-1.13.0 through pg-jev, for about $0.30 in API calls in total.

Finding	Result
Scores are over-confident	9-way choice ECE 0.089; yes/no ECE 0.24–0.38
~100 labels fix most of it	Calibration on 100 labels removes 45–85 % of the error
Asking again changes answers	4.8 % of rows change top answer across 3 identical runs
Neighbours change answers	Re-batching the same rows changes ~16.5 % of answers (≈ 5× noise)
One row per request is better	Judging rows singly: +2.4 pp accuracy, +0.05 macro-F1, at 1.8× tokens
Adding an option moves answers	2.5 % of answers move to a new catch-all option
Scores are coarse	Probabilities come on a 0.01 grid with exact zeros

Write-up for a general audience: Same row, different answer. Details, methods and limits: docs/phase0/FINDINGS-jev-1.13.md.

Install

pip install "git+https://github.com/ElyasEbrahim/pgdocket"      # Python 3.10+; PyPI release coming
pgdocket --version

Or as a container: docker build -t pgdocket . from this repository. New here? The 5-minute quickstart runs Postgres, the worker and 60 synthetic support tickets with Docker. It costs about $0.001 in API calls.

Try it on your own data: pgdocket check

Point it at a table, the text to send, a question and some labels (an existing column, or a CSV you label by hand). It reports agreement, calibration, a threshold with a stated precision guarantee, consistency and cost. It's read-only, and no row text goes in the report.

pgdocket check --table public.tickets --key id --text subject,body \
  --kind choice --question "Which team should handle this ticket?" \
  --options billing,technical,security,sales --label-column team --yes

On two years of Kubernetes issues, routing to 10 teams (SIGs) agreed with the human labels 88.8 % of the time. 90 % of issues could be auto-routed with precision ≥ 90 % guaranteed at 95 % confidence (write-up, report). Full usage: docket/README.md.

What pgdocket does, and will do

1. Stored decisions. Built. Answers live in an append-only ledger and a view you can join and index. New and changed rows are judged automatically, one row per request. 100,000 rows: one row reads in 0.14 ms; on a live stream, the median time from insert to answer was 0.7 s (details).
2. Calibration on your labels. Built. About 100–200 human labels per decision turn raw scores into probabilities that mean what they say. The view shows the calibrated confidence and an auto flag.
3. Honest numbers. Built. Thresholds come with stated guarantees ("precision ≥ 90 % at 95 % confidence"). Counts come with confidence intervals: pgdocket estimate combines the model's answers with a random sample of labels (prediction-powered inference). On real Kubernetes issues its intervals held 98–100 % of the time and were 30–50 % narrower than labels alone (details).
4. Versions. Partly built. Changing the question, labels or model creates a new version; comparing versions before switching is not built yet.
5. Runs anywhere. Built and tested on PostgreSQL 14–18 as a non-superuser. Plain SQL plus an external worker, no untrusted extension. Not yet tested on Supabase, Neon or RDS themselves.

Architecture and SQL sketch: REPORT.md §4. Decisions so far: docs/adr/.

Quick start: the Phase 0 harness

Requires Docker. Everything runs in containers.

make up            # Postgres 16 + pg-jev v0.2.1, pg-jev's mock API, a simulated backend
make test          # harness unit tests
make smoke         # pg-jev's README queries against the mock
make fetch load    # 2,000 CFPB complaint narratives (CC0 mirror) → Postgres
make bench         # all experiments on the simulated backend → results/<run_id>/report.md

To run against the real Jev API, copy .env.example to .env, add your TypeSafe key, then:

make bench BACKEND=jev ARGS=--yes   # prints the estimated spend; refuses anything over $1

More commands (determinism, learning curve, batch context): harness/README.md.

Repository map

Path	What
REPORT.md	Project report: pg-jev teardown, research survey, candidates, design, evaluation plan, risks
ROADMAP.md	Phased plan with exit criteria
CHANGELOG.md	What changed, by date
docs/adr/	Architecture decision records
docs/phase0/	Phase 0 findings, status, draft upstream issue
docs/notes/	Public write-ups
docs/phase1/	Phase 1 results: the ledger, counts with error bars, Kubernetes triage
docket/	The product: pgdocket check (and later the ledger)
examples/	The 5-minute quickstart, and option files for the example checks
pyproject.toml, Dockerfile	Packaging: the pgdocket command and the worker image (releasing)
harness/	Benchmark harness: jevbench CLI, simulated backend, tests
results/	Benchmark reports from the runs cited in the findings
docker-compose.yml, Makefile	The benchmark stack and its commands

Sources

• realZachi/pg-jev, the Postgres extension this builds on (PostgreSQL License).
• winkrun, 不用向量索引，Postgres 也能做语义过滤：pg-jev 上手 (4 Oct 2026). Its list of pg-jev caveats shaped the design; the article isn't redistributed here.
• Full reference list: REPORT.md § Sources.

Contributing

This is a single-maintainer project. Issues and questions are welcome; pull requests are not accepted for now. See CONTRIBUTING.md.

License

Apache-2.0. Copyright 2026 ElyasEbrahim.