Adding Evaluation suite to test hardware and Network by rushabhdhoke · Pull Request #3135 · dimensionalOS/dimos

rushabhdhoke · 2026-07-22T03:19:09Z

Contribution path

Created a folder "eval" inside the "dimos" folder
root/dimos/eval/

Problem

Requirement of an FDE-facing eval utility for Jetson, Nvidia, AGX systems. Added Network eval for simulating network stability for real world deployments.

Solution

Introduced "dimos/eval` with two standalone tools that reusing existing DimOS infrastructure.

Resource eval - uses the StatsMonitor (used by dtop) to an in-memory logger to calculate startup cost. Measures CPU, PSS Memory, GPU utilization and startup times for Jetson and Nividia hardware. Outputs a JSON file with a hardware profile.
Network eval - Uses the same measure_throughput function as the existing benchmark then runs the 'tc netem'. It evaluates the transport, profilepair and calculates % loss and latency.

How to Test

Resource eval — list available blueprints

python -m dimos.eval --list

Network eval (smoke test)

python -m dimos.eval.network --baseline-only

Full netem sweep (Linux + root )

sudo -E python -m dimos.eval.network --iface lo

AI assistance

Gemini 3.5 Flash & Claude Sonnet 4.6

Checklist

I have read and approved the CLA.

greptile-apps · 2026-07-22T03:21:12Z

Greptile Summary

This PR adds hardware and network evaluation tools for DimOS. The main changes are:

Hardware detection and resource sampling for blueprint runs.
JSON and terminal reports for CPU, memory, GPU, and startup metrics.
Shared pubsub throughput measurement under network impairment profiles.
Command-line entry points for resource and network evaluations.

Confidence Score: 5/5

The latest changes look safe to merge.

The interval checks prevent non-positive values from starting tight sampler loops.
Monitor cleanup now runs when evaluation fails after startup.
No additional blocking issue met the requirements for this follow-up review.

Important Files Changed

Filename	Overview
dimos/eval/network.py	Adds transport selection, network impairment profiles, result grading, reporting, and CLI handling.
dimos/eval/runner.py	Adds blueprint execution, resource monitoring, result aggregation, interval validation, and cleanup.
dimos/eval/measure.py	Adds the shared pubsub throughput measurement used by network evaluations.
dimos/eval/sampler.py	Adds in-memory process sampling, GPU polling, and resource aggregation.
dimos/eval/hardware.py	Adds host, Jetson, CUDA, and GPU hardware detection.
dimos/eval/metrics.py	Adds evaluation result models, JSON serialization, and terminal reports.
dimos/eval/main.py	Adds the command-line interface for blueprint resource evaluation.

_{Reviews (4): Last reviewed commit: "Merge branch 'main' into main" | Re-trigger Greptile}

greptile-apps · 2026-07-22T03:21:15Z

+
+def _display_name(case: Any) -> str:
+    """Pretty transport label, shared with the benchmark's pubsub_id."""
+    from dimos.protocol.pubsub.benchmark.measure import transport_name


Missing Benchmark Module

Both _display_name() and run_network_eval() import helpers from dimos.protocol.pubsub.benchmark.measure, but that module is absent from the repository. The documented network run fails with ModuleNotFoundError before measuring a transport, and --list fails when it calls _display_name().

module.py was existing in previous dimos versions, current version doesn't have that file, so this was failing. Added the file in my folder and referenced it.

greptile-apps · 2026-07-22T03:21:16Z

+
+def networked_cases() -> list[Any]:
+    """The subset of the benchmark's testcases that actually cross a link."""
+    from dimos.protocol.pubsub.benchmark.testdata import testcases


Runtime Path Requires Pytest

networked_cases() imports the benchmark's test-data module, which imports pytest even though it is only a test dependency. A normal installation without development dependencies therefore fails when a network run or --list reaches this line.

Wrapped the from testdata import testcases in a try/except ImportError that gives a clear error message telling the user to install dev dependencies.

greptile-apps · 2026-07-22T03:21:17Z

+        worker_source = cast("WorkerManagerPython", coordinator._managers["python"])
+        monitor = StatsMonitor(worker_source, resource_logger=resource_logger, interval=interval)
+
+        monitor.start()


Failed Runs Leak Monitor Thread

After this monitor starts, an exception during GPU startup, warmup, sampling, or aggregation skips monitor.stop(). The finally block stops the GPU and coordinator but leaves this daemon monitor running, so repeated failed API calls accumulate sampling threads.

Added monitor.stop() to the finally block. The monitor variable is now tracked in the outer scope (initialized to None) so finally can always reach it, even if the exception happens after monitor.start().

greptile-apps · 2026-07-22T03:21:18Z

+        build_start = time.monotonic()
+        from dimos.core.coordination.module_coordinator import ModuleCoordinator
+
+        coordinator = ModuleCoordinator.build(bp)


Partial Build Loses Cleanup Handle

ModuleCoordinator.build() starts managers before completing deployment. If a later build step raises, this assignment never stores the coordinator, so the finally block sees None and cannot stop the workers already started by the failed build.

Changed the build to built = ModuleCoordinator.build(bp) then coordinator = built on the next line. If build() raises partway through, coordinator stays None so finally won't try to call .stop() on a half-built object. The actual teardown of any partially-started managers is ModuleCoordinator's responsibility.

greptile-apps · 2026-07-22T03:21:19Z

+                        help="Steady-state sampling window in seconds (default: 15).")
+    parser.add_argument("--warmup", type=float, default=3.0,


Zero Interval Corrupts Measurements

The CLI accepts --interval 0, which makes both sampler threads repeatedly call Event.wait(0) in a tight loop. Their CPU usage then becomes part of the benchmark, producing inflated CPU results while consuming a core.

Fixed in runner.py + main.py, run_eval() now raises ValueError if interval <= 0
The CLI in main.py validates --interval > 0 before calling run_eval()

Adjusted formatting of the 'status' field in the EvalResult dataclass for consistency.

greptile-apps · 2026-07-23T04:34:05Z

+        from dimos.protocol.pubsub.benchmark.testdata import testcases
+    except ImportError as exc:
+        raise ImportError(
+            "The transport benchmark test-data module requires pytest and its "
+            "dependencies to be installed. Install with: uv sync --all-extras"
+        ) from exc


Runtime dependency remains unresolved

Importing benchmark.testdata still executes its unconditional import pytest. A normal installation without test dependencies therefore cannot run --list or any network evaluation. This handler only replaces the original exception, and its suggested uv sync --all-extras command does not install the test group containing pytest. Move the reusable cases out of the pytest module or declare the required runtime dependency.

greptile-apps · 2026-07-23T04:34:06Z

+        built = ModuleCoordinator.build(bp)
+        coordinator = built


Partial builds still leak

ModuleCoordinator.build() starts worker processes before connecting streams and references or starting modules. If one of those later steps raises, build() never returns, so coordinator remains None. The finally block then skips coordinator.stop(), leaving workers or partially started modules running. Cleanup must happen inside ModuleCoordinator.build() for every exception after startup, or the coordinator must be available before those fallible steps run.

rushabhdhoke added 3 commits July 21, 2026 21:49

Add initial markdown file in eval directory

11ef912

Add files via upload

116044a

Delete dimos/eval/.md

3b12af2

rushabhdhoke requested review from leshy, mustafab0, paul-nechifor and spomichter as code owners July 22, 2026 03:19

greptile-apps Bot reviewed Jul 22, 2026

View reviewed changes

autofix-ci Bot and others added 10 commits July 22, 2026 03:39

[autofix.ci] apply automated fixes

265e901

Update __init__.py

d574b8c

Refactor argument parsing for clarity and conciseness

7e553b1

Remove empty line in HardwareProfile class

7a59917

Refactor 'status' field formatting in EvalResult

2ae4880

Adjusted formatting of the 'status' field in the EvalResult dataclass for consistency.

Add files via upload

85ee742

Refactor imports and update argument parsing

1092a49

Fix docstrings and improve error handling in runner.py

719d5f0

Refactor aggregate_worker function parameters formatting

7ed9f4f

Update README with evaluation suite details

903cfbd

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Merge branch 'main' into main

d86c621

		help="Steady-state sampling window in seconds (default: 15).")
		parser.add_argument("--warmup", type=float, default=3.0,

Uh oh!

Conversation

rushabhdhoke commented Jul 22, 2026

Contribution path

Problem

Solution

How to Test

Resource eval — list available blueprints

Network eval (smoke test)

Full netem sweep (Linux + root )

AI assistance

Checklist

Uh oh!

greptile-apps Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

greptile-apps Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

rushabhdhoke Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

rushabhdhoke Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

rushabhdhoke Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

rushabhdhoke Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

rushabhdhoke Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Jul 22, 2026 •

edited

Loading