Skip to content

Compare TabPFN-3 with the Random Forest on the same synthetic split - #3

Merged
kpernyer merged 4 commits into
mainfrom
examples/tabpfn-comparison
Sep 8, 2026
Merged

Compare TabPFN-3 with the Random Forest on the same synthetic split#3
kpernyer merged 4 commits into
mainfrom
examples/tabpfn-comparison

Conversation

@kpernyer

@kpernyer kpernyer commented Sep 8, 2026

Copy link
Copy Markdown

Compare a pinned pretrained TabPFN-3 classifier with the existing Random Forest and historical base rate on exactly the same fictional commitment snapshots. This builds on the Random Forest example in #2.

The optional CUDA adapter verifies the checkpoint hash, checks matching training/test identities and exclusions, and predicts one test row per call using fixed historical training context. The report adds checkpoint/code provenance, per-row probabilities, library/device versions, timing, and peak PyTorch allocation. Weights and run artefacts stay outside Git. No service is deployed or GPU CI runner configured.

The guide documents the isolated environment, exact checkpoint revision and non-commercial model terms, bounded allocation, and limitations of a single synthetic comparison. No organisational validity or broad model-superiority claim is made.

Validation: 25 local tests pass, including checkpoint tampering, positive-class mapping, valid probabilities, and single-row prediction boundaries. The existing 11-event contract fixture passes. The real pinned checkpoint ran successfully on an NVIDIA GB10 with PyTorch 2.11.0+cu130; all 25 tests also passed there. On the same 153 training / 85 test synthetic rows, TabPFN achieved Brier 0.1500 and log loss 0.4760 versus the forest's 0.1677 and 0.5200. Accuracy at 0.5 was 77.65% versus 78.82%. The guide records the measured result, source commit, dataset hash, environment, and limitations. The existing inference service remained healthy.

@kpernyer
kpernyer changed the base branch from examples/random-forest-baseline to main September 8, 2026 17:19
@kpernyer
kpernyer merged commit 74e6ea2 into main Sep 8, 2026
6 checks passed
@kpernyer
kpernyer deleted the examples/tabpfn-comparison branch September 8, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant