Skip to content

Support AWS p6 instances#3961

Merged
r4victor merged 4 commits into
masterfrom
issue_3940_aws_p6_support
Jun 15, 2026
Merged

Support AWS p6 instances#3961
r4victor merged 4 commits into
masterfrom
issue_3940_aws_p6_support

Conversation

@r4victor

@r4victor r4victor commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Closes #3940

Add support for AWS p6-b200 and p6-b300 instances. Specifically, update EFA configuration logic as every p6 type requires custom configuration (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html)

Also update EFA Docker image to include latest EFA-related software that support Blackwell. The current image didn't work.

NCCL tests on 2x p6-200:

#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)

           8             2     float     sum      -1    50.58    0.00    0.00       0    54.46    0.00    0.00       0
          16             4     float     sum      -1    54.55    0.00    0.00       0    54.52    0.00    0.00       0
          32             8     float     sum      -1    54.59    0.00    0.00       0    55.34    0.00    0.00       0
          64            16     float     sum      -1    55.60    0.00    0.00       0    55.51    0.00    0.00       0
         128            32     float     sum      -1    56.27    0.00    0.00       0    56.26    0.00    0.00       0
         256            64     float     sum      -1    57.14    0.00    0.01       0    56.54    0.00    0.01       0
         512           128     float     sum      -1    54.89    0.01    0.02       0    53.72    0.01    0.02       0
        1024           256     float     sum      -1    56.52    0.02    0.03       0    55.72    0.02    0.03       0
        2048           512     float     sum      -1    57.42    0.04    0.07       0    58.40    0.04    0.07       0
        4096          1024     float     sum      -1    65.08    0.06    0.12       0    62.18    0.07    0.12       0
        8192          2048     float     sum      -1    62.92    0.13    0.24       0    65.11    0.13    0.24       0
       16384          4096     float     sum      -1    63.52    0.26    0.48       0    63.80    0.26    0.48       0
       32768          8192     float     sum      -1    68.55    0.48    0.90       0    63.88    0.51    0.96       0
       65536         16384     float     sum      -1    65.94    0.99    1.86       0    68.60    0.96    1.79       0
      131072         32768     float     sum      -1    69.73    1.88    3.52       0    72.51    1.81    3.39       0
      262144         65536     float     sum      -1    88.11    2.98    5.58       0    88.38    2.97    5.56       0
      524288        131072     float     sum      -1    92.30    5.68   10.65       0    86.44    6.07   11.37       0
     1048576        262144     float     sum      -1    95.76   10.95   20.53       0    97.25   10.78   20.22       0
     2097152        524288     float     sum      -1   107.75   19.46   36.49       0   110.37   19.00   35.63       0
     4194304       1048576     float     sum      -1   138.53   30.28   56.77       0   138.33   30.32   56.85       0
     8388608       2097152     float     sum      -1   154.79   54.19  101.61       0   154.86   54.17  101.57       0
    16777216       4194304     float     sum      -1   190.75   87.95  164.91       0   188.86   88.84  166.57       0
    33554432       8388608     float     sum      -1   292.61  114.67  215.01       0   291.07  115.28  216.15       0
    67108864      16777216     float     sum      -1   481.39  139.40  261.38       0   481.32  139.43  261.43       0
   134217728      33554432     float     sum      -1   744.86  180.19  337.86       0   744.39  180.31  338.07       0
   268435456      67108864     float     sum      -1  1209.14  222.00  416.26       0  1198.05  224.06  420.11       0
   536870912     134217728     float     sum      -1  2401.34  223.57  419.20       0  2352.59  228.20  427.88       0
  1073741824     268435456     float     sum      -1  3742.18  286.93  537.99       0  3725.04  288.25  540.47       0
  2147483648     536870912     float     sum      -1  6696.75  320.68  601.27       0  6694.57  320.78  601.46       0
  4294967296    1073741824     float     sum      -1  12714.6  337.80  633.37       0  12653.1  339.44  636.45       0
  8589934592    2147483648     float     sum      -1  24589.9  349.33  654.99       0  24577.6  349.50  655.32       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 144.893
#
# Collective test concluded: all_reduce_perf
#

NCCL tests on 2x p6-300:

           8             2     float     sum      -1    52.59    0.00    0.00       0    55.36    0.00    0.00       0
          16             4     float     sum      -1    55.83    0.00    0.00       0    55.54    0.00    0.00       0
          32             8     float     sum      -1    63.49    0.00    0.00       0    57.03    0.00    0.00       0
          64            16     float     sum      -1    56.09    0.00    0.00       0    56.66    0.00    0.00       0
         128            32     float     sum      -1    58.28    0.00    0.00       0    57.42    0.00    0.00       0
         256            64     float     sum      -1    57.46    0.00    0.01       0    59.40    0.00    0.01       0
         512           128     float     sum      -1    57.19    0.01    0.02       0    59.40    0.01    0.02       0
        1024           256     float     sum      -1    58.32    0.02    0.03       0    58.76    0.02    0.03       0
        2048           512     float     sum      -1    61.75    0.03    0.06       0    61.44    0.03    0.06       0
        4096          1024     float     sum      -1    63.86    0.06    0.12       0    63.24    0.06    0.12       0
        8192          2048     float     sum      -1    67.63    0.12    0.23       0    66.69    0.12    0.23       0
       16384          4096     float     sum      -1    66.20    0.25    0.46       0    68.05    0.24    0.45       0
       32768          8192     float     sum      -1    69.04    0.47    0.89       0    71.83    0.46    0.86       0
       65536         16384     float     sum      -1    71.60    0.92    1.72       0    70.87    0.92    1.73       0
      131072         32768     float     sum      -1    73.05    1.79    3.36       0    72.36    1.81    3.40       0
      262144         65536     float     sum      -1    87.76    2.99    5.60       0    87.54    2.99    5.62       0
      524288        131072     float     sum      -1    90.57    5.79   10.85       0    91.32    5.74   10.76       0
     1048576        262144     float     sum      -1    97.55   10.75   20.16       0    97.00   10.81   20.27       0
     2097152        524288     float     sum      -1   109.56   19.14   35.89       0   110.80   18.93   35.49       0
     4194304       1048576     float     sum      -1   139.79   30.00   56.26       0   141.73   29.59   55.49       0
     8388608       2097152     float     sum      -1   153.14   54.78  102.71       0   155.10   54.08  101.41       0
    16777216       4194304     float     sum      -1   189.00   88.77  166.44       0   193.26   86.81  162.77       0
    33554432       8388608     float     sum      -1   283.43  118.39  221.98       0   283.00  118.57  222.31       0
    67108864      16777216     float     sum      -1   394.06  170.30  319.31       0   392.35  171.04  320.71       0
   134217728      33554432     float     sum      -1   596.39  225.05  421.97       0   597.03  224.81  421.51       0
   268435456      67108864     float     sum      -1  1023.87  262.18  491.58       0  1025.24  261.83  490.93       0
   536870912     134217728     float     sum      -1  1988.76  269.95  506.16       0  1990.46  269.72  505.73       0
  1073741824     268435456     float     sum      -1  3910.37  274.59  514.85       0  3910.56  274.57  514.83       0
  2147483648     536870912     float     sum      -1  5606.43  383.04  718.20       0  5615.47  382.42  717.04       0
  4294967296    1073741824     float     sum      -1  10827.2  396.68  743.78       0  10911.1  393.63  738.06       0
  8589934592    2147483648     float     sum      -1  21327.9  402.76  755.17       0  21412.5  401.16  752.18       0

# Out of bounds values : 0 OK
# Avg bus bandwidth    : 164.191
#

@r4victor r4victor merged commit c21b686 into master Jun 15, 2026
24 checks passed
@r4victor r4victor deleted the issue_3940_aws_p6_support branch June 15, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support AWS B200 (p6-b200) Instances

1 participant