Skip to content

fix(machine-controller): report DPU restart outcome accurately#3904

Open
behroozrafii wants to merge 1 commit into
NVIDIA:mainfrom
behroozrafii:behrooz/3713-dpu-restart-logging
Open

fix(machine-controller): report DPU restart outcome accurately#3904
behroozrafii wants to merge 1 commit into
NVIDIA:mainfrom
behroozrafii:behrooz/3713-dpu-restart-logging

Conversation

@behroozrafii

Copy link
Copy Markdown
Contributor

DPU restart handling logged success and queued the reboot-request timestamp before the Redfish restart completed. When the restart failed, logs could falsely imply success, and the lower-level failure log did not identify the affected DPU.

This change:

  • Logs the restart attempt before calling Redfish.
  • Logs failures with both dpu_machine_id and the returned error.
  • Records the reboot-request timestamp and logs success only after the restart succeeds.
  • Removes the redundant lower-level failure log while preserving the original error.

This gives operators an accurate restart outcome and identifies the failed DPU without changing any API or configuration.

Related issues

Fixes #3713

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)
  • Existing unit tests run:
    • cargo test -p carbide-machine-controller --lib
    • 56 passed, 0 failed, 0 ignored, 0 measured, 0 filtered out
  • Formatting verified:
    • cargo fmt --all -- --check

Additional Notes

No API, configuration, or database-schema changes.

Signed-off-by: Behrooz Rafii <brafii@nvidia.com>
@behroozrafii
behroozrafii requested a review from a team as a code owner July 23, 2026 01:34
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e5cac7ed-e6fa-4b21-af12-18ab02d40046

📥 Commits

Reviewing files that changed from the base of the PR and between 862b66a and 4fa5d42.

📒 Files selected for processing (1)
  • crates/machine-controller/src/handler.rs

Summary by CodeRabbit

  • Bug Fixes
    • DPU restart requests now update reboot status only after the restart succeeds.
    • Failed restart attempts are reported immediately without recording an incorrect reboot-request timestamp.
    • Improved error handling for restart failures.

Walkthrough

The DPU restart handler now performs the Redfish restart before recording reboot-request state or logging success. Restart errors propagate directly, preventing database updates and success logs when the operation fails.

Changes

DPU restart sequencing

Layer / File(s) Summary
Restart execution and success-gated persistence
crates/machine-controller/src/handler.rs
restart_dpu directly propagates Redfish restart errors, while handler_restart_dpu records UpdateRebootRequestedTime and logs success only after a successful restart.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title concisely describes the main change: making DPU restart outcome reporting accurate.
Description check ✅ Passed The description matches the code changes and clearly explains the restart logging and timestamp behavior.
Linked Issues check ✅ Passed The implementation matches #3713 by logging attempts before restart, deferring success and DB updates until after success, and preserving failure context.
Out of Scope Changes check ✅ Passed The changes stay within the restart-handling scope and do not introduce unrelated behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: DPU reprovision restart logs success before Redfish restart is sent

1 participant