fix(#584): _is_object_storage recognises data_access_protocol=='object'

[FileSystemGuy]

Summary

Resolves #584.

validation_helpers._is_object_storage(args) decided object-vs-local using only two signals:

1. --params storage.storage_type=s3 present in args.params
2. s3://-scheme prefix on args.data_dir / args.checkpoint_folder

It did not consult args.data_access_protocol == 'object' — the canonical signal set by the object positional, and the signal every other site already keys on (CAP-01 gate per #568/#579, dlio.py, run_summary, cli_parser, every CLI module).

For a bare mlpstorage … run object --data-dir data/unet3d …:

• data_access_protocol='object' is set at parse time ✓
• storage.storage_type=s3 is injected onto DLIOBenchmark.params_dict after parsing (mlpstorage_py/benchmarks/dlio.py:202-203), so it never reaches args.params at validation time
• the bare --data-dir has no s3:// scheme

_is_object_storage returned False, _validate_paths fell through to os.path.exists() on what is conceptually a bucket key, and the run died with [E401] Data directory not found. Only --skip-validation could unblock it, which also disables the MPI / SSH / DLIO checks — too broad.

The fix

One line, additive — check data_access_protocol first, then fall through to the legacy signals. Every config that returned True before still does.

def _is_object_storage(args) -> bool:
    if getattr(args, 'data_access_protocol', None) == 'object':
        return True
    # … existing legacy signals unchanged …

Tests

12 new tests, all pass; full unit suite stays green.

TestIsObjectStorage (9 cases):

• canonical signal (data_access_protocol='object') → True (the _is_object_storage() misses the object positional (data_access_protocol) → object-mode runs fail [E401] Data directory not found / checkpoint parent-dir checks #584 regression lock)
• data_access_protocol='file' → False (must not misclassify file mode)
• missing data_access_protocol attr → falls through to legacy signals
• legacy --params storage.storage_type=s3 → True (and =object variant)
• s3:// on data_dir → True
• s3:// on checkpoint_folder → True
• unrelated --params entries don't flip the gate
• empty Namespace() doesn't raise

TestValidatePathsObjectStorageBypass (3 cases — integration through _validate_paths):

• exact _is_object_storage() misses the object positional (data_access_protocol) → object-mode runs fail [E401] Data directory not found / checkpoint parent-dir checks #584 reproducer config (object positional, bare relative data_dir, no --params) → no errors
• guardrail: file mode with a missing data_dir still errors (the bypass must not leak)
• checkpoint parent-dir check is also bypassed for object mode (the second site the reporter mentioned)

Verification

• uv run pytest tests/unit/test_validation_helpers.py -v → 46 passed (34 existing + 12 new).
• uv run pytest tests/unit -q → 2303 passed. No regressions.
• On-cluster: reporter's repro (mlpstorage closed training unet3d run object --data-dir data/unet3d …, no --skip-validation) should now pass environment validation.

Notes

• Independent of fix(#568): refine CAP-01 object-storage skip — key on storage_type, add helper + dedicated tests #581. That PR refines a different _is_object_storage — DLIOBenchmark._is_object_storage() in benchmarks/dlio.py, scoped to the CAP-01 capacity gate with access to params_dict. Same name, different scope. They could converge into a shared helper eventually, but _is_object_storage() misses the object positional (data_access_protocol) → object-mode runs fail [E401] Data directory not found / checkpoint parent-dir checks #584 is the user-facing CLI-validation bug and is solved in isolation here.
• Independent of fix(#566,#569): CAP-02 shared-FS probe works on FUSE; preserve scp basename #577 (CAP-02 / scp fixes for CAP-02 shared-FS probe rejects all multi-host DAOS/FUSE filesystems due to spurious st_dev comparison #566+Cluster collector remote staging hardcodes the destination filename, breaking the multi-host CAP-02 shared-FS probe #569) and MERGE ALL EARLIER PRs FIRST -- chore: bump dlio-benchmark to DLIO #39; version 3.0.25 -> 3.0.26 #582 (DLIO Single host reportgen not working? #39 pin bump for --o-direct / storage_type=direct_fs crashes on local NPZ/NPY/JPEG — storage_library sentinel disagreement between config validation ("s3dlio") and the iterable reader ("direct") #567). All three rebase cleanly regardless of merge order.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[russfellows]

Not sure all this is needed to determine if we have object storage, but Ok.