[aw471]

DO NOT REVIEW / WIP

Add an experimental, opt-in executor for distributing one KumoTabular estimator across GPUs.

• Shard expanded row embeddings and retained context KV caches; distribute fit attention heads and complete query batches.
• Add owner-local frontend execution, bounded per-owner row workspaces, and explicit FP16 execution while retaining FP32 defaults.
• Add collective failure handling, fitted-state lifecycle checks, regression coverage, and a runnable example.
• Keep research reports and benchmark provenance outside the library branch.

Remaining before review: resolve the current-main conflict in sdm/models/kumo/tabular/row_embedding.py, run the complete suite on the final integrated revision, and complete final-revision CUDA lifecycle and numerical validation. This draft is not a production-readiness claim.

Preliminary GPU results

Single eight-B200 host, FP16, one estimator, owner-local frontend and sharded row workspaces enabled. Both datasets use 500 prepared features and 512 validation queries in batches of eight.

Dataset / context rows	GPUs	Fit median	Predict median	Peak allocated per GPU
EPSILON / 262,144	1	20.73 s	11.61 s	77.16 GiB
EPSILON / 262,144	2	11.37 s	6.18 s	41.32 GiB
EPSILON / 262,144	4	6.06 s	4.13 s	25.36 GiB
EPSILON / 262,144	8	3.29 s	3.65 s	17.38 GiB
URL / 2,097,152	4	166.15 s	23.66 s	144.38 GiB
URL / 2,097,152	8	85.10 s	11.90 s	80.53 GiB

EPSILON achieves 6.31× fit and 3.18× prediction speedup from one to eight GPUs. Complete predictions are bit-identical across GPU counts and the native FP16 reference.

URL exceeds one billion context cells. Native one-GPU and two-owner execution run out of memory; four and eight GPUs complete. Doubling GPUs gives 1.95× fit and 1.99× prediction speedup, with bit-identical predictions between those configurations. No successful single-GPU numerical reference exists at this size.

Measurement limits: One run per configuration, three fits and nine measured predictions after excluding prediction warmups. Memory includes warmups and repeated fits. These are not cold-start speedups: EPSILON first-fit time is 26.44 s on one GPU versus 22.53 s on eight. Aggregate physical memory increases despite lower per-GPU peaks—79.92→214.80 GiB for EPSILON and 599.84→695.11 GiB for URL.

Measurements used the implementation before the final FP32-backend compatibility adjustment. The final PR revision still needs GPU validation. These results demonstrate workload-specific scaling, not universal linear capacity or production readiness.

Scaling efficiency

Parallel efficiency = observed speedup ÷ increase in GPU count. At a fixed workload, 100% means ideal linear scaling.

Dataset	GPUs	Ideal speedup	Fit speedup / efficiency	Prediction speedup / efficiency
EPSILON	1→2	2×	1.82× / 91.2%	1.88× / 93.9%
EPSILON	1→4	4×	3.42× / 85.6%	2.81× / 70.2%
EPSILON	1→8	8×	6.31× / 78.9%	3.18× / 39.8%
EPSILON	4→8	2×	1.84× / 92.2%	1.13× / 56.7%
URL	4→8	2×	1.95× / 97.6%	1.99× / 99.4%

Scaling interpretation: These measurements demonstrate near-linear fixed-context speed scaling for URL from four to eight GPUs—not universal linear scaling. EPSILON fit achieves 78.9% efficiency across one to eight GPUs, while prediction achieves only 39.8% and shows diminishing returns beyond four GPUs.

First-fit latency scales differently: EPSILON’s first fit improves only 1.17× from one to eight GPUs; URL’s first fit improves 1.71× from four to eight. The larger speedups in the table describe the reported repeated-call medians, not process cold-start.

Context capacity is a separate claim: URL completes on four/eight GPUs where one/two fail, demonstrating increased capacity. We have not established that maximum supported context grows linearly with GPU count. Whole-workflow per-GPU allocation also falls less than ideally: 1.79× for URL when doubling GPUs.

Each configuration has one measured run with repeated calls, not independent-run confidence intervals. The final PR revision still requires GPU validation.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.