Add Hot Aisle bare metal support

[peterschmidt85]

Hot Aisle bare metal (experimental)

Opt in with bare_metal: true:

projects:
- name: main
  backends:
  - type: hotaisle
    team_handle: hotaisle-team-handle
    creds:
      type: api_key
      api_key: ...
    bare_metal: true

$ dstack offer --backend hotaisle
 #  BACKEND                   RESOURCES                 INSTANCE TYPE  PRICE
 1  hotaisle (us-michigan-1)  cpu=x86:104 mem=2048GB    bm-mi300x-8    $27.12
                              disk=115335GB
                              gpu=MI300X:192GB:8

Bare metal servers are prepaid for 8 hours. Use a fleet with fixed nodes and delete it when done:

type: fleet
name: hotaisle-bm

nodes: 1
backends: [hotaisle]
instance_types: [bm-mi300x-8]

Within the 8 hours, dstack doesn't force-terminate bare metal servers: the instance stays terminating for about 15 minutes, then dstack stops tracking it and the server must be terminated manually in Hot Aisle. Controlled by DSTACK_FF_HOTAISLE_BARE_METAL_NO_FORCE_RELEASE (enabled by default, 0 disables).

Instance names

Hot Aisle instance types are renamed to vm-mi300x-<count> and bm-mi300x-8. The CPU varies by server but the price doesn't, so it's not in the name.

Other changes

• Starting dstack-shim over SSH is retried until it succeeds, instead of a single unchecked attempt.

Notes

• Bare metal SSH goes through ssh_access; ip_address is private.
• The OS is on a 438 GiB root disk; the 8x 15.36TB NVMe drives are mounted at /mnt/nvme*n1p1 (reachable via instance volumes). The offer's disk size is still the total of all drives.

Follow-ups

• Finish the real bare metal test.
• gpuhunt: report the boot disk as bare metal disk_size: the M.2 480GB entry in Specs.disks (the root filesystem is 438 GiB), instead of disk_capacity (115,335 GB, the total of all 10 drives).
• Docs: mention the 8x 15.36TB NVMe drives mounted at /mnt/nvme{0,1,2,3,5,6,7,8}n1p1 and how to use them via instance volumes, e.g. volumes: [/mnt/nvme0n1p1:/data].
• Keep instances idle until their minimum reservation period ends: gpuhunt adds MinimumReservationMinutes to the offer's provider_data, the backend sets a new JobProvisioningData.reserved_until on create, and the idle-termination check skips instances before reserved_until. Then drop DSTACK_FF_HOTAISLE_BARE_METAL_NO_FORCE_RELEASE.

🤖 Generated with Claude Code