Compare TabPFN-3 with the Random Forest on the same synthetic split

.github/workflows/check.yml

@@ -25,3 +25,18 @@ jobs:
       - run: python -m unittest discover -s tests -v
       - run: python -m com_jepa validate examples/fictional-trajectory.jsonl
       - run: python -m com_jepa snapshot examples/fictional-trajectory.jsonl --as-of 2026-01-06T12:00:00Z
+
+  forest-demo:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: '3.14'
+      - run: python -m pip install -r requirements-ml.txt
+      - run: python -m unittest discover -s tests -v
+      - run: python -m com_jepa validate examples/fictional-trajectory.jsonl
+      - run: python -m com_jepa.forest_demo --output artifacts/forest-demo.json

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased — initial research scaffold
 
+- Add an optional scikit-learn Random Forest demo on explicitly invented tabular snapshots, with a training-only base-rate comparator, temporal label eligibility, exclusion accounting, reproducible reports, and ML CI. No real organisational findings are implied.
 - Establish the organisational thesis, claims ledger, and baseline-first research plan.
 - Propose a draft `0.1.0` commitment event export with version lineage and occurrence/availability timestamps.
 - Add a wholly fictional trajectory, context projection utility, and temporal/semantic validation tests.

README.md

@@ -8,7 +8,7 @@ Today, people often carry the missing connections between business systems. They
 
 **com-jepa asks whether a history of commitments, decisions, actions, and observed consequences can improve the next organisational decision.** It starts with an explicit data contract and simple statistical models. It will investigate Joint Embedding Predictive Architectures (JEPA) if the evidence and data justify that step.
 
-This is an open research project initiated by [Reflective Lab](https://www.reflective.se). It is at the research-design and data-contract stage. There is no trained organisational JEPA model, production predictor, or empirical performance claim in this repository. The included data is entirely fictional.
+This is an open research project initiated by [Reflective Lab](https://www.reflective.se). It is at the research-design and data-contract stage, with an executable Random Forest demonstration trained on invented rows. There is no trained organisational JEPA model, production predictor, or empirical organisational performance claim in this repository. The included data is entirely fictional.
 
 [Reflective research library](https://www.reflective.se/labs/research) · [Research plan](docs/research-plan.md) · [Data contract](docs/data-contract.md) · [Contribute](CONTRIBUTING.md) · [Security](SECURITY.md)
 
@@ -68,6 +68,8 @@ A simple model that wins is a successful research result. A shared model valid f
 | [Foundations](docs/foundations.md) | The organisation, human agency, and learning from priors |
 | [Research plan](docs/research-plan.md) | Hypotheses, comparisons, evaluation, and stop conditions |
 | [Data contract](docs/data-contract.md) | What to collect, label, connect, and keep out |
+| [Random Forest demo](docs/random-forest-demo.md) | A small CPU-only classifier, historical base-rate comparison, and temporal checks on synthetic data |
+| [TabPFN-3 comparison](docs/tabpfn-comparison.md) | An optional GPU experiment using the same rows and a pinned pretrained classifier |
 | [Application integration](docs/application-integration.md) | How apps contribute to the Organisation Core |
 | [First 90 days](docs/first-90-days.md) | A bounded starting project and collaboration questions |
 | [Reading list](docs/reading-list.md) | Intellectual lineage and the limits of the evidence |
@@ -90,11 +92,22 @@ The validation command reports **11 valid fictional events**. The Tuesday-noon s
 
 ## Project status
 
+To try the optional ML example, use Python 3.14 and the pinned scikit-learn dependencies:
+
+```sh
+python3.14 -m venv .venv
+.venv/bin/python -m pip install -r requirements-ml.txt
+.venv/bin/python -m com_jepa.forest_demo
+```
+
+This fits a Random Forest to 300 invented commitment snapshots and compares it with the training fulfilment rate on a later test period. Unknown/disputed/censored labels are excluded, and training labels must be available before fitting. See the [demo guide](docs/random-forest-demo.md) for the exact synthetic rule, JSON reports, and limitations. PyTorch and a GPU are not required.
+
 | Available now | Proposed next |
 | --- | --- |
 | Organisational thesis and falsifiable research questions | Practitioner review and selection of one commitment family |
 | Draft JSON Schema, fictional trajectory, and historical context utility | A governed prospective pilot and real outcome adjudication |
 | Temporal and lineage tests; GitHub CI | A reproducible benchmark with conventional ML baselines |
+| Synthetic Random Forest example and base-rate comparator | Real-data feature extraction, outcome adjudication, and validation |
 | Evaluation and application-integration proposals | Sequence/graph experiments and, if justified, JEPA |
 
 No partner participation, dataset access, generalisation result, or model efficiency is implied by this roadmap. See the [claims ledger](docs/reading-list.md#claim-boundaries).

com_jepa/forest_demo.py

@@ -0,0 +1,225 @@
+"""CPU-only Random Forest demonstration on explicitly fictional tabular snapshots.
+
+Run: python -m com_jepa.forest_demo --output artifacts/forest-demo.json
+This is a pipeline exercise, not a validated organisational predictor.
+"""
+
+import argparse
+from collections import Counter
+from datetime import datetime, timedelta, timezone
+from hashlib import sha256
+from importlib.metadata import version
+import json
+import math
+from pathlib import Path
+import random
+
+from .events import timestamp
+
+
+FEATURES = ("days_remaining", "unresolved_dependencies", "evidence_age_days", "remaining_work_units")
+TARGET = "all_criteria_met_by_version_deadline"
+SETTLED = {"met": 1, "not_met": 0}
+UNRESOLVED = {"unknown", "right_censored", "disputed"}
+GENERATOR_VERSION = "toy-tabular-v1"
+START = datetime(2025, 1, 1, tzinfo=timezone.utc)
+
+
+def stamp(value):
+    return value.strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def generate_rows(samples=300, seed=42):
+    """Invent one snapshot per independent initiative, with a declared noisy rule.
+
+    Numbers and causal-looking relationships are authored assumptions. This does
+    not consume the event ledger or assign labels to the discussion cards.
+    """
+    if not 120 <= samples <= 2000:
+        raise ValueError("samples must be between 120 and 2000 for this small demo")
+    rng = random.Random(seed)
+    rows = []
+    for i in range(samples):
+        as_of = START + timedelta(days=i)
+        days, dependencies, age, work = rng.randint(2, 28), rng.randint(0, 4), rng.randint(0, 10), rng.randint(2, 30)
+        logit = 1.2 + 0.13 * days - 0.75 * dependencies - 0.06 * age - 0.13 * work
+        probability = 1 / (1 + math.exp(-logit))
+        status = "met" if rng.random() < probability else "not_met"
+        if rng.random() < 0.06:
+            status = rng.choice(sorted(UNRESOLVED))
+        due = as_of + timedelta(days=days)
+        rows.append({
+            "data_origin": "fictional", "generator_version": GENERATOR_VERSION,
+            "organisation_id": "fictional-org", "initiative_id": f"initiative-{i:04d}",
+            "commitment_id": f"commitment-{i:04d}", "commitment_version": 1,
+            "target": TARGET, "as_of": stamp(as_of), "due_at": stamp(due),
+            "features_available_at": stamp(as_of),
+            "features": dict(zip(FEATURES, (days, dependencies, age, work))),
+            "label_status": status,
+            "label_available_at": stamp(due + timedelta(days=rng.randint(1, 5))),
+        })
+    return rows
+
+
+def feature_matrix(rows):
+    """Explicit allowlist: neither identities nor future outcome fields are inputs."""
+    matrix = []
+    for row in rows:
+        if timestamp(row["features_available_at"]) > timestamp(row["as_of"]):
+            raise ValueError("features were unavailable at prediction cutoff")
+        if set(row["features"]) != set(FEATURES):
+            raise ValueError("unexpected or missing feature; review the allowlist")
+        values = [row["features"][name] for name in FEATURES]
+        if any(isinstance(v, bool) or not isinstance(v, (float, int)) or not math.isfinite(v) or v < 0 for v in values):
+            raise ValueError("features must be finite non-negative numbers")
+        days = (timestamp(row["due_at"]) - timestamp(row["as_of"])).total_seconds() / 86400
+        if days <= 0 or days != row["features"]["days_remaining"]:
+            raise ValueError("days_remaining must match the future version deadline")
+        matrix.append(values)
+    return matrix
+
+
+def temporal_split(rows, training_cutoff, test_start, evaluation_as_of):
+    """One snapshot per initiative; train labels must have settled before fitting.
+
+    This narrow contract deliberately rejects repeated initiatives rather than
+    pretending to implement full trajectory/component grouping.
+    """
+    train_at, test_at, evaluate_at = map(timestamp, (training_cutoff, test_start, evaluation_as_of))
+    if not train_at < test_at <= evaluate_at:
+        raise ValueError("require training_cutoff < test_start <= evaluation_as_of")
+    feature_matrix(rows)
+    seen_initiatives, seen_commitments = set(), set()
+    train, test, excluded = [], [], Counter()
+    for row in rows:
+        if row["data_origin"] != "fictional" or row["generator_version"] != GENERATOR_VERSION:
+            raise ValueError("this demonstration accepts only its declared fictional row format")
+        if row["target"] != TARGET or row["commitment_version"] != 1:
+            raise ValueError("this demo supports only the original version and fixed target")
+        initiative = row["organisation_id"], row["initiative_id"]
+        commitment = row["organisation_id"], row["commitment_id"]
+        if initiative in seen_initiatives or commitment in seen_commitments:
+            raise ValueError("demo requires one unique initiative and commitment per row")
+        seen_initiatives.add(initiative)
+        seen_commitments.add(commitment)
+        as_of, known = timestamp(row["as_of"]), timestamp(row["label_available_at"])
+        if known < timestamp(row["due_at"]):
+            raise ValueError("assessment availability cannot precede the deadline in this demo")
+        status = row["label_status"]
+        if status not in SETTLED and status not in UNRESOLVED:
+            raise ValueError("unsupported label status")
+        if as_of < train_at:
+            if known > train_at:
+                excluded["training_label_not_yet_available"] += 1
+            elif status not in SETTLED:
+                excluded[f"training_{status}"] += 1
+            else:
+                train.append(row)
+        elif as_of < test_at:
+            excluded["temporal_gap"] += 1
+        elif as_of > evaluate_at or known > evaluate_at:
+            excluded["test_not_yet_assessable"] += 1
+        elif status not in SETTLED:
+            excluded[f"test_{status}"] += 1
+        else:
+            test.append(row)
+    order = lambda r: (r["as_of"], r["initiative_id"])
+    return sorted(train, key=order), sorted(test, key=order), dict(sorted(excluded.items()))
+
+
+def run_demo(samples=300, seed=42):
+    # Optional dependency: the event-validation commands do not import sklearn.
+    from sklearn.dummy import DummyClassifier
+    from sklearn.ensemble import RandomForestClassifier
+    from sklearn.metrics import accuracy_score, brier_score_loss, log_loss
+
+    rows = generate_rows(samples, seed)
+    training_cutoff = stamp(START + timedelta(days=int(samples * 0.6)))
+    test_start = stamp(timestamp(training_cutoff) + timedelta(days=30))
+    evaluation_as_of = stamp(START + timedelta(days=samples + 40))
+    train, test, excluded = temporal_split(rows, training_cutoff, test_start, evaluation_as_of)
+    x_train, x_test = feature_matrix(train), feature_matrix(test)
+    y_train = [SETTLED[r["label_status"]] for r in train]
+    y_test = [SETTLED[r["label_status"]] for r in test]
+    if len(set(y_train)) != 2 or not y_test:
+        raise ValueError("need both training classes and at least one settled test outcome")
+    models = {
+        "historical_base_rate": DummyClassifier(strategy="prior"),
+        "random_forest": RandomForestClassifier(
+            n_estimators=100, max_depth=6, min_samples_leaf=5, random_state=seed, n_jobs=1,
+        ),
+    }
+    scores, probabilities = {}, {}
+    for name, model in models.items():
+        model.fit(x_train, y_train)
+        positive_column = list(model.classes_).index(1)
+        predicted = model.predict_proba(x_test)[:, positive_column]
+        probabilities[name] = predicted.tolist()
+        scores[name] = {
+            "brier_score": float(brier_score_loss(y_test, predicted)),
+            "log_loss": float(log_loss(y_test, predicted, labels=[0, 1])),
+            "accuracy_at_0_5": float(accuracy_score(y_test, predicted >= 0.5)),
+        }
+    payload = json.dumps(rows, sort_keys=True, separators=(",", ":"), allow_nan=False)
+    return {
+        "purpose": "synthetic_pipeline_demonstration_not_organisational_evidence",
+        "generator_version": GENERATOR_VERSION, "seed": seed,
+        "target": TARGET, "positive_class": "met",
+        "features": list(FEATURES), "dataset_sha256": sha256(payload.encode()).hexdigest(),
+        "environment": {p: version(p) for p in ("scikit-learn", "numpy", "scipy")},
+        "training_cutoff": training_cutoff, "test_start": test_start,
+        "evaluation_as_of": evaluation_as_of,
+        "counts": {"generated": len(rows), "train": len(train), "test": len(test), "excluded": excluded},
+        "train_fulfilment_rate": sum(y_train) / len(y_train),
+        "test_fulfilment_rate": sum(y_test) / len(y_test),
+        "training_initiatives": [r["initiative_id"] for r in train],
+        "forest_parameters": models["random_forest"].get_params(),
+        "metrics": scores,
+        "test_predictions": [
+            {"initiative_id": row["initiative_id"], "as_of": row["as_of"],
+             "label_available_at": row["label_available_at"], "observed_label": row["label_status"],
+             "features": row["features"],
+             **{name: values[i] for name, values in probabilities.items()}}
+            for i, row in enumerate(test)
+        ],
+        "limitations": [
+            "Labels follow an invented rule, not observations from organisations.",
+            "One synthetic organisation; independent initiatives; no revisions, dependencies between rows, or interventions.",
+            "Same generator in train and test; no real-world transfer or JEPA conclusions.",
+            "Fixed forest; no tuning, calibration study, uncertainty intervals, or causal claims.",
+            "Toy snapshots are not exports of the draft event ledger or the fictional challenge cards.",
+        ],
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--samples", type=int, default=300)
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--output", type=Path, help="optional JSON report; use ignored artifacts/")
+    args = parser.parse_args()
+    try:
+        report = run_demo(args.samples, args.seed)
+        if args.output:
+            args.output.parent.mkdir(parents=True, exist_ok=True)
+            with args.output.open("x") as stream:
+                json.dump(report, stream, indent=2, allow_nan=False)
+                stream.write("\n")
+    except ModuleNotFoundError:
+        parser.exit(1, "Install the optional demo dependencies: python -m pip install -r requirements-ml.txt\n")
+    except (ValueError, OSError) as error:
+        parser.exit(1, f"error: {error}\n")
+    print("SYNTHETIC PIPELINE DEMO — not evidence of organisational prediction quality")
+    print(f"Rows: {report['counts']['generated']} generated; {report['counts']['train']} train; {report['counts']['test']} test")
+    print(f"Train labels available by {report['training_cutoff']}; test snapshots from {report['test_start']}")
+    print(f"Excluded: {json.dumps(report['counts']['excluded'], sort_keys=True)}")
+    print("Model                     Brier (lower)  Log loss (lower)  Accuracy @ 0.5")
+    for name, metrics in report["metrics"].items():
+        print(f"{name:25} {metrics['brier_score']:.4f}         {metrics['log_loss']:.4f}            {metrics['accuracy_at_0_5']:.4f}")
+    print("Random Forest probabilities are uncalibrated; no causal or transfer claim.")
+    if args.output:
+        print(f"Report: {args.output}")
+
+
+if __name__ == "__main__":
+    main()