Skip to content

[DirectVerbs] Implement send-related interfaces in the dv transport #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pakmarkthub wants to merge 62 commits into
base: devel-dv
Choose a base branch
from
Open

[DirectVerbs] Implement send-related interfaces in the dv transport #106

pakmarkthub wants to merge 62 commits into from

Conversation

@pakmarkthub
Copy link
Contributor

@pakmarkthub pakmarkthub commented Oct 27, 2021

Prerequisite:

This PR:

  • implements init_send_info, prepare_send, get_num_send_request_entries, post_send_ops, post_send_ops_on_cpu, get_send_descs transport interfaces in mlx5-dv.cpp.
  • refactors gds_post_ops and gds_post_ops_on_cpu.
  • refactors gds_mlx5_get_send_descs.

Presubmit testing:

  • on gc11 (for mlx5-dv transport).
  • on Summit (for mlx5-exp transport).

@pakmarkthub
Added --with-spectrum-mpi to configure
ce78eb6
@pakmarkthub
Introduced GDS_DRIVER_TYPE to gds_qp and gds_cq
66409ac
@pakmarkthub
Set gds_qp and gds_cq dtype to MLX5_EXP in the creation functions
fdd2270
@pakmarkthub
Changed the definition of gds_qp_init_attr_t and gds_send_wr. Also, r…
0073f81 
…emoved res_domain from the gds_qp struct
@pakmarkthub
Implemented gds_get_driver_type
78eadbb
@pakmarkthub
Initial implementation of mlx5-exp.cpp/hpp and moved create/destroy q…
aade2ab 
…p/cq functions to mlx5-exp
@pakmarkthub
Implemented gds_mlx5_exp_destroy_cq
5e789db
@pakmarkthub
Implemented gds_destroy_qp and gds_destroy_cq by connecting to gds_ml…
2a8ac7d 
…x5_exp_*
@pakmarkthub
Reimplemented gds_create_qp by connecting to gds_mlx5_exp_create_qp
a9351f2
@pakmarkthub
Moved gds_send_wr from exp-verbs to ib-verbs in gds_kernel_latency.c
e188668
@pakmarkthub
Fixed compile issues in gdsync.cpp and mlx5-exp.hpp
07d03a0
@pakmarkthub
Added mlx5-exp.cpp to the compile list
173841f
@pakmarkthub
Moved gds_prepare_send to gds_mlx5_exp_* and fixed compile errors in …
8d1ed50 
…mlx5-exp.*
@pakmarkthub
Modified gds_kernel_* applications to fit the new API/structs
10e164e
@pakmarkthub
Moved gds_wait_request to gds_mlx5_exp_wait_request and made the form…
b1cab05 
…er an opaque struct
@pakmarkthub
Moved gds_wait_request related functions to mlx5-exp
17b246f
@pakmarkthub
Changed the definition of gds_send_request_t
6be27ac
@pakmarkthub
Added gds_mlx5_exp_send_request_t definition and supported functions
f026578
@pakmarkthub
Moved APIs that use gds_send_request_t to mlx5-exp.cpp
92b09f0
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from public header files
e088b01
@pakmarkthub
Replaced structs related to ibv_exp_ with gds_* structs
942aa89
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from objs.hpp
35db866
@pakmarkthub
Removed include verbs_exp.h from objs.cpp
24a0a20
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from gdsync.cpp
5d7c328
@pakmarkthub
Relaced all IBV_EXP_ enum with GDS_ in gdsync.cpp
bf1a358
@pakmarkthub
Replaced all IBV_EXP_ enum with GDS_ in mlx5.cpp
f95c72d
@pakmarkthub
Relaced all IBV_EXP_ enum with GDS_ enum in gdsync_debug_hostregister…
6a23931 
…_bug.cpp
@pakmarkthub
Fixed bugs in gds_dump_wait_request
ab745de
@pakmarkthub
Updated some functions in gdsync_debug_hostregister_bug.cpp to reflec…
f43f905 
…t changes in gdsync.cpp
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from all files except from…
aeb1ed7 
… mlx-exp.*
@pakmarkthub
Fixed bugs and modified the code to use the gds_transport_t interface
de7d52d
@pakmarkthub
Removed driver_type and fixed bugs
c914e89
@pakmarkthub
Added exp-verbs support checking in transport.hpp
ed703cc
@pakmarkthub
Fixed typo in configure.ac
7340483
@pakmarkthub
Fixed compile errors when compiling on x86
9986794
@pakmarkthub
Added initial implementation of DEVX and DirectVerbs QP/CQ create/des…
1ea8492 
…troy
@pakmarkthub
Connected dv destroy_qp to the transport interface
1c302b4
@pakmarkthub
Introduced gds_modify_qp
97a5f81
@pakmarkthub
Fixed typo in the help text in gds_kernel_latency
c053bb3
@pakmarkthub
Implemented CQ buffer allocation on GPU memory using DirectVerbs
d4f1f41
@pakmarkthub
Added UD support to QP creation with DEVX
f0c5167
@pakmarkthub
Implemented gds_mlx5_dv_post_recv
b99b28c
@pakmarkthub
Moved test applications to use gds_post_recv
3b51bd9
@pakmarkthub
Implemented gds_mlx5_exp_post_recv
e8235ae
@pakmarkthub
Fixed bugs when running with exp verbs
834e896
@pakmarkthub
Fixed bugs in gds_mlx5_dv_modify_qp when using UD connection
803d670
@pakmarkthub
Merge branch 'dev-dv-post-recv' of github.com:pakmarkthub/libgdsync i…
072e754 
…nto dev-dv-post-recv
@pakmarkthub
Defined data structures related to gds_mlx5_dv_send_request_t
22156ae
@pakmarkthub
Implemented gds_mlx5_dv_init_send_info
db4bcd9
@pakmarkthub
Implemented gds_mlx5_dv_post_send with limited support
d3fcaac
@pakmarkthub
Implemented gds_mlx5_dv_peer_commit_qp and gds_mlx5_dv_prepare_send
123e9d2
@pakmarkthub
Fixed bug in gds_mlx5_dv_post_wrs
d78329d
@pakmarkthub
Implemented gds_mlx5_dv_get_num_send_request_entries
4ee99fe
@pakmarkthub
Moved gds_post_ops and gds_post_ops_on_cpu back to gdsync.cpp
82b62cb
@pakmarkthub
Implemented gds_mlx5_dv_post_send_ops
f576435
@pakmarkthub
Fixed bugs in mlx5-exp.cpp
2e21e7f
@pakmarkthub
Implemented gds_mlx5_dv_post_send_ops_on_cpu
254932c
@pakmarkthub
Implemented gds_mlx5_dv_get_send_descs
5b2e2c2
@pakmarkthub pakmarkthub requested a review from drossetti October 27, 2021 06:21
@pakmarkthub pakmarkthub mentioned this pull request Nov 12, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drossetti drossetti Awaiting requested review from drossetti

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pakmarkthub