Skip to content

[DirectVerbs] Implement post recv in the dv transport#105

Open
pakmarkthub wants to merge 51 commits intogpudirect:devel-dvfrom
pakmarkthub:dev-dv-post-recv
Open

[DirectVerbs] Implement post recv in the dv transport#105
pakmarkthub wants to merge 51 commits intogpudirect:devel-dvfrom
pakmarkthub:dev-dv-post-recv

Conversation

@pakmarkthub
Copy link
Contributor

@pakmarkthub pakmarkthub commented Oct 22, 2021

Prerequisite:

This PR:

  • introduces post_recv transport interface.
  • introduces gds_mlx5_dv_post_recv.
  • introduces gds_mlx5_exp_post_recv.
  • changes gds_post_recv implementation to call post_recv transport interface.
  • fixes bugs when running with exp verbs.
  • modifies the test applications to use gds_post_recv instead of directly calling ibv_post_recv.

Presubmit testing:

  • on gc11 and on Summit.

@pakmarkthub
Added --with-spectrum-mpi to configure
ce78eb6
@pakmarkthub
Introduced GDS_DRIVER_TYPE to gds_qp and gds_cq
66409ac
@pakmarkthub
Set gds_qp and gds_cq dtype to MLX5_EXP in the creation functions
fdd2270
@pakmarkthub
Changed the definition of gds_qp_init_attr_t and gds_send_wr. Also, r…
0073f81 
…emoved res_domain from the gds_qp struct
@pakmarkthub
Implemented gds_get_driver_type
78eadbb
@pakmarkthub
Initial implementation of mlx5-exp.cpp/hpp and moved create/destroy q…
aade2ab 
…p/cq functions to mlx5-exp
@pakmarkthub
Implemented gds_mlx5_exp_destroy_cq
5e789db
@pakmarkthub
Implemented gds_destroy_qp and gds_destroy_cq by connecting to gds_ml…
2a8ac7d 
…x5_exp_*
@pakmarkthub
Reimplemented gds_create_qp by connecting to gds_mlx5_exp_create_qp
a9351f2
@pakmarkthub
Moved gds_send_wr from exp-verbs to ib-verbs in gds_kernel_latency.c
e188668
@pakmarkthub
Fixed compile issues in gdsync.cpp and mlx5-exp.hpp
07d03a0
@pakmarkthub
Added mlx5-exp.cpp to the compile list
173841f
@pakmarkthub
Moved gds_prepare_send to gds_mlx5_exp_* and fixed compile errors in …
8d1ed50 
…mlx5-exp.*
@pakmarkthub
Modified gds_kernel_* applications to fit the new API/structs
10e164e
@pakmarkthub
Moved gds_wait_request to gds_mlx5_exp_wait_request and made the form…
b1cab05 
…er an opaque struct
@pakmarkthub
Moved gds_wait_request related functions to mlx5-exp
17b246f
@pakmarkthub
Changed the definition of gds_send_request_t
6be27ac
@pakmarkthub
Added gds_mlx5_exp_send_request_t definition and supported functions
f026578
@pakmarkthub
Moved APIs that use gds_send_request_t to mlx5-exp.cpp
92b09f0
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from public header files
e088b01
@pakmarkthub
Replaced structs related to ibv_exp_ with gds_* structs
942aa89
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from objs.hpp
35db866
@pakmarkthub
Removed include verbs_exp.h from objs.cpp
24a0a20
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from gdsync.cpp
5d7c328
@pakmarkthub
Relaced all IBV_EXP_ enum with GDS_ in gdsync.cpp
bf1a358
@pakmarkthub
Replaced all IBV_EXP_ enum with GDS_ in mlx5.cpp
f95c72d
@pakmarkthub
Relaced all IBV_EXP_ enum with GDS_ enum in gdsync_debug_hostregister…
6a23931 
…_bug.cpp
@pakmarkthub
Fixed bugs in gds_dump_wait_request
ab745de
@pakmarkthub
Updated some functions in gdsync_debug_hostregister_bug.cpp to reflec…
f43f905 
…t changes in gdsync.cpp
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from all files except from…
aeb1ed7 
… mlx-exp.*
@pakmarkthub
Updated configure.ac and Makefile.am to make exp-verbs optional
59d20f5
@pakmarkthub
Fixed bug in configure.ac related to checking for exp-verbs
bfb683b
@pakmarkthub
Defined gds_transport_t
e61a405
@pakmarkthub
Moved mlx5-exp.* into transports/mlx5-exp and implemented gds_transpo…
8720c86 
…rt_mlx5_exp_init
@pakmarkthub
Fixed bugs and modified the code to use the gds_transport_t interface
de7d52d
@pakmarkthub
Removed driver_type and fixed bugs
c914e89
@pakmarkthub
Added exp-verbs support checking in transport.hpp
ed703cc
@pakmarkthub
Fixed typo in configure.ac
7340483
@pakmarkthub
Fixed compile errors when compiling on x86
9986794
@pakmarkthub
Added initial implementation of DEVX and DirectVerbs QP/CQ create/des…
1ea8492 
…troy
@pakmarkthub
Connected dv destroy_qp to the transport interface
1c302b4
@pakmarkthub
Introduced gds_modify_qp
97a5f81
@pakmarkthub
Fixed typo in the help text in gds_kernel_latency
c053bb3
@pakmarkthub
Implemented CQ buffer allocation on GPU memory using DirectVerbs
d4f1f41
@pakmarkthub
Added UD support to QP creation with DEVX
f0c5167
@pakmarkthub
Implemented gds_mlx5_dv_post_recv
b99b28c
@pakmarkthub
Moved test applications to use gds_post_recv
3b51bd9
@pakmarkthub
Implemented gds_mlx5_exp_post_recv
e8235ae
@pakmarkthub
Fixed bugs when running with exp verbs
834e896
@pakmarkthub pakmarkthub requested a review from drossetti October 22, 2021 07:35
@pakmarkthub pakmarkthub mentioned this pull request Oct 27, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drossetti drossetti Awaiting requested review from drossetti

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pakmarkthub