Skip to content

Keep runtime configuration validation active under python -O - #8573

Open
Saisharathchandranandnetha wants to merge 3 commits into
deepspeedai:masterfrom
Saisharathchandranandnetha:fix/config-validation-optimized-mode
Open

Saisharathchandranandnetha wants to merge 3 commits into
deepspeedai:masterfrom
Saisharathchandranandnetha:fix/config-validation-optimized-mode

Conversation

@Saisharathchandranandnetha

Copy link
Copy Markdown

Fixes #8572.

Running Python with -O removes the assertions used by DeepSpeedConfig, allowing invalid batch sizes, inconsistent batch arithmetic, missing batch settings, and incompatible precision options to bypass validation. Replace those assertions with explicit checks and raise AssertionError so validation remains active while preserving the existing exception type, conditions, and error messages.

Regression coverage exercises zero, negative, and NaN batch values; inconsistent batch arithmetic; simultaneous fp16/bf16; missing batch settings; and fp16/bf16 ZeRO requirements. A subprocess regression runs actual optimized Python during a normal test run.

Validation:

  • python -m pytest tests/unit/runtime/test_ds_config_dict.py -q -k "TestConfigValidation": 16 passed.
  • The same command with python -O: 16 passed.
  • Final focused runs including the optimized-Python subprocess regression: 17 passed in each mode.
  • Full test file in both modes: 54 passed, 4 skipped, 12 environment failures. The failures occur in multiprocessing forkserver setup because the execution sandbox blocks Unix-socket binding, before the affected test bodies run. The optimized-Python subprocess regression passes.
  • Test environment: Intel Core i5-12450HX CPU, Python 3.14.7, PyTorch 2.14.0+cu130; no accessible GPU. Distributed/GPU integration remains unverified.
  • All applicable repository pre-commit hooks passed for the two changed files. Used a temporary config with identical hooks and Python 3.11 as the hook interpreter because pinned flake8 5.0.4 crashes on Python 3.14; the repository hook configuration is unchanged.
  • git diff --check: passed.

Signed-off-by: Saisharathchandranandnetha <saisharathchandranandnetha@gmail.com>
Signed-off-by: Saisharathchandranandnetha <saisharathchandranandnetha@gmail.com>
Ensure configuration validation tests fail deterministically regardless of the distributed WORLD_SIZE environment variable in CI.

Signed-off-by: DeepSpeed Contributor <contributor@deepspeed.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Runtime configuration validation is bypassed under python -O

1 participant