Skip to content

[vocabTP] Add masked streamed log-probabilities and entropy - #9117

Draft
jinyouzhi wants to merge 1 commit into
deepspeedai:masterfrom
jinyouzhi:vocabtp-streamed-scoring
Draft

jinyouzhi wants to merge 1 commit into
deepspeedai:masterfrom
jinyouzhi:vocabtp-streamed-scoring

Conversation

@jinyouzhi

Copy link
Copy Markdown
Contributor

Summary

Add vocab_parallel_linear_logprobs, an opt-in primitive that computes FP32
per-token selected log-probabilities and optional entropy directly from hidden
states and resident vocabulary-sharded LM-head parameters.

Refs #8790 and the third request in
#8942 (comment).

  • Select scored positions before projection, skipping prompt/tool-output
    positions in the LM head while retaining them as backbone context.
  • Stream token and local-vocabulary tiles, normalize across TP ranks and
    recompute projection tiles in backward without retaining full local logits.
  • Support bias, temperature, ignored positions, all-masked/empty batches and
    uneven/empty vocabulary shards. Return unreduced scores; the caller owns
    causal alignment, masking policy and loss.
  • Add dense-oracle, gradient/update, Engine and TP tests, plus tutorial/API docs.

Scope and related work

Existing vocab-parallel scoring ops consume materialized logits; fused-linear
CE does not expose selected-token scores and entropy. This complements the
sequence chunking in deepspeedai/DeepSpeedExamples#1023 and the rollout/GRPO
work in #9037; it does not change either architecture or implement a GRPO loss.
Duplicate searches found no existing streamed-scoring PR.

Hidden gradients are reduced inside this primitive exactly once: pass original
decoder hidden states, not the head's TP backward-reduction mapping.
An opt-in allow_empty_shards argument extends the existing metadata helper;
existing callers keep their previous behavior.

Validation

Hardware: 2 x NVIDIA L20, physical GPUs 0 and 3, driver 580.173.02.
Software: Python 3.10.22, PyTorch 2.14.1+cu130, Transformers 5.17.0,
pytest 9.1.1.

Test Result
New scoring suite: 23 CPU cases plus 12 CUDA Engine/TP cases 35 passed
Existing fused-linear CE and vocab-parallel ops regression 140 passed, 10 skipped
Scoped pre-commit and git diff --check Passed
export DS_ACCELERATOR=cuda CUDA_VISIBLE_DEVICES=0,3
export PYTHONPATH=tests:. OMP_NUM_THREADS=2
python -m pytest --forked \
  tests/unit/v1/sequence_parallelism/test_vocab_parallel_linear_logprobs.py -q
python -m pytest --forked \
  tests/unit/v1/sequence_parallelism/test_vocab_parallel_linear_cross_entropy.py \
  tests/unit/v1/sequence_parallelism/test_vocab_parallel_ops.py -q

CUDA coverage includes TP2 FP32/BF16 scores and hidden/head/bias gradients,
uneven/empty shards, all-masked/zero-token batches and collective input errors.
A TP1 ZeRO-0 Engine test matches dense loss, gradients and parameters over
three AdamW updates. Additional FP32 integration tests cover:

  • An actual HF Llama with AutoTP2, tied embedding/head, uneven vocabulary
    shards, and ZeRO-0/2. Over three SGD updates, scores, reconstructed shared
    gradients/weights and final-normalization gradients/weights match the dense
    reference. The test verifies that AutoTP preserves the shared Parameter.
  • DP2 ZeRO-2 with communication overlap and GAS2. Ranks use different inputs,
    including an all-masked microbatch on one rank. Full averaged gradients and
    parameters match the dense reference over three AdamW updates.

The 10 skips are existing JSD cases whose world-size list includes four ranks;
only the requested two GPUs were visible. Both test commands completed
successfully: 94.71 seconds for the final new suite, 291.60 seconds for regression.

Draft limitations

Only resident linear heads are supported; ZeRO-3 parameters are rejected.
No automatic trainer integration, sequence-sharded hidden states, quantized
heads, higher-order gradients or implicit model-specific/sampling transforms.
The distribution is full-vocabulary softmax after temperature; matching rollout
scores requires a separately agreed probability contract.

Bounded projection tiles do not bound total memory: compaction retains the scored
hidden states, and head gradients accumulate in a full local-shard FP32 buffer.
Eager tile launches, per-token-chunk collectives and input-validation
synchronization are additional costs that need end-to-end measurement.

Not qualified: new scoring with FP16 loss scaling, BF16 ZeRO-2 integration,
ZeRO-1, offload, complete external GRPO/TRL consumers, other accelerators,
or combined TP x DP, PP/SP/EP.
Full-step throughput, peak memory, token capacity and convergence have not
been benchmarked; no improvement in those metrics is claimed. Those
measurements and a real consumer remain follow-ups before ready-for-review.

Potential follow-ups (not implemented in this PR)

These are separate directions for discussion and qualification, not claims of
support or commitments to expand this PR:

  • A real consumer and end-to-end measurements. Coordinate a minimal
    integration with Add a multi-turn GRPO example with tool calling, and port OPSD to the per-sample rollout API DeepSpeedExamples#1023, keeping rollout and GRPO
    loss in the example. Compare full-step throughput, peak memory and token
    capacity against its existing sequence-chunked head and a sequence-chunked
    vocab-parallel baseline; validate training numerics before claiming a benefit.
  • Generic scoring parity with TRL's fused head. Use TRL commit
    0ac80058
    as the initial contract reference. Potential additions include logit scaling
    and soft-capping in the correct order, TP-global support_ids normalization,
    multiple score_ids, log_sum_sq_probs, mean_logits, is_top1, and an
    explicit FP32 projection option. Match padding, tie-breaking, numerical
    stability and differentiable/non-differentiable output semantics, with dense
    and TRL-reference output/gradient tests at TP1/TP2.
  • A downstream TRL adapter. Keep model patching, causal or pre-shifted label
    alignment, loss normalization, MoE auxiliary loss and PEFT/generation behavior
    in TRL or its integration layer, rather than in this Core primitive. Backend
    selection must be TP-aware: an unsupported path must not silently normalize
    over only a local vocabulary shard. Full TRL fused-head compatibility is not
    provided by the current primitive.
  • Parameter lifecycle and compatibility qualification. Investigate ZeRO-3
    head materialization/release across forward and recomputed backward.
    Separately qualify BF16 ZeRO-2 integration, FP16 loss scaling, offload and
    combined parallel layouts; define contracts for non-plain heads before
    claiming support.
  • Shared helpers and additional distillation consumers. Keep this primitive
    independent initially; consider consolidating common streamed-projection
    machinery when more consumers justify it. Streamed dense KL/JSD would be a
    separate primitive, potentially relevant to
    Add vocabulary-parallel KL support to OPSD DeepSpeedExamples#1019: selected-token scores and entropy cannot
    replace its exact full-vocabulary distillation loss. Profile the actual
    bottleneck before prioritizing that extension.

Prepared with GitHub Copilot CLI assistance.

…k support

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant