Add data-loading bottleneck diagnostic skill

[rostan-t]

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Create a skill allowing agent to detect and localize data loading bottlenecks in PyTorch training.

The workflow roughly contains two important phases:

• Detection of whether or not there's a data loading bottleneck using DALI's LoaderEvaluator
• Bottleneck localization with profiling

The complete workflow is more complex and looks as follows:

1. Identify the workload, environment and if it's at all possible to run.
2. Run (bounded) training, measure loader wait. If LoaderEvaluator is or can be made importable , continue to 3. Else, go to 4.
3. Replay the training with cached batches using LoaderEvaluator's replay mode. Establish if there's a measured bottleneck. If we this steps provdes that there's no bottleneck, go to 5. Otherwise, continue to 4. to localize.
4. Run profiling to localize the bottleneck if we come from 3., or to detect it if coming from 2.
5. Report the results.

This was tested on multiple models, with different environment requirements, including simple ResNet50, OpenCLIP, LeWorldModel, and Hugging Face timm.

In terms of the model used by the agent, I found that GPT 5.6 Terra or equivalent is a sound minimum requirement to run this reliably, although it can sometimes fail to accurately follow all instructions, like the shape of the report.

I unfortunately couldn't commit evaluations as the skill CI doesn't support GPU environments.

Additional information:

Affected modules and functionalities:

Skill created.

Key points relevant for the review:

• Is the skill's workflow sound?
• Can the skill miss an existing bottlenecks or detect one where there isn't?
• Is the skill too defensive on things like requiring equivalent work for replay?

Note that if you run locally, depending on the workload, it may take time to complete.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Skill CI doesn't support GPU workers yet

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other (skill)
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4796

[greptile-apps]

Greptile Summary

The PR adds a skill for diagnosing data-loading bottlenecks in CUDA-enabled PyTorch training.

• Defines bounded Real-versus-Replay measurement and verdict thresholds.
• Documents Nsight Systems profiling, validation, localization, and reporting workflows.
• Adds preflight and profile-summary scripts plus a structured report template.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
skills/data-loading-bottleneck/SKILL.md	Defines the end-to-end bounded detection, replay, profiling, and reporting workflow.
skills/data-loading-bottleneck/references/profiling.md	Documents profile instrumentation, structural validation, overhead checks, and evidence-based localization.
skills/data-loading-bottleneck/references/pytorch-dali.md	Specifies construction and equivalence requirements for paired Real and Replay loaders.
skills/data-loading-bottleneck/scripts/collect_preflight.py	Collects workload, environment, GPU, source, storage, and optional-tool readiness information.
skills/data-loading-bottleneck/scripts/summarize_nsys.py	Exports and summarizes bounded NVTX and CUDA activity from Nsight Systems reports.
skills/data-loading-bottleneck/assets/report-template.md	Provides the required structure for workload, detection, localization, recommendation, and confidence reporting.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Preflight workload and environment] --> B{Canonical run available?}
  B -- No --> Z[Report INCONCLUSIVE]
  B -- Yes --> C[Run bounded Real workload]
  C --> D{LoaderEvaluator available?}
  D -- Yes --> E[Run equivalent Replay workload]
  E --> F{Replay speedup}
  F -- <= 1.10x --> G[Report NOT DETECTED]
  F -- > 1.10x --> H[Profile and localize]
  D -- No --> H
  H --> I{Validated input-path delay?}
  I -- Yes --> J[Report DETECTED or POTENTIAL with cause]
  I -- No --> Z

Loading

_{Reviews (2): Last reviewed commit: "Add data-loading bottleneck diagnostic s..." | Re-trigger Greptile}

[JanuszL]

/nvskills-ci