Skip to content

rdma/siw: implement non-blocking connect

Stefan Metzmacher linux-rdma 2 versions New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest14 patches 0 repliesNew
  2. v17 patches 4 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/14rdma/siw: remove superfluous siw_cep_put() from siw_connect() error pathdrivers/infiniband/sw/siw/siw_cm.c+0 -10 replies
2/14rdma/siw: make siw_cm_upcall() a noop without valid 'id'drivers/infiniband/sw/siw/siw_cm.c+3 -00 replies
3/14rdma/siw: split out a __siw_cep_terminate_upcall() functiondrivers/infiniband/sw/siw/siw_cm.c+32 -210 replies
4/14rdma/siw: use __siw_cep_terminate_upcall() for indirect SIW_CM_WORK_CLOSE_LLPdrivers/infiniband/sw/siw/siw_cm.c+1 -50 replies
5/14rdma/siw: use __siw_cep_terminate_upcall() for SIW_CM_WORK_PEER_CLOSEdrivers/infiniband/sw/siw/siw_cm.c+50 -390 replies
6/14rdma/siw: use __siw_cep_terminate_upcall() for SIW_CM_WORK_MPATIMEOUTdrivers/infiniband/sw/siw/siw_cm.c+8 -230 replies
7/14rdma/siw: handle SIW_EPSTATE_CONNECTING in __siw_cep_terminate_upcall()drivers/infiniband/sw/siw/siw_cm.c+8 -10 replies
8/14rdma/siw: make use of kernel_{bind,connect,listen}()drivers/infiniband/sw/siw/siw_cm.c+7 -70 replies
9/14rdma/siw: let siw_connect() set AWAIT_MPAREP before siw_send_mpareqrep()drivers/infiniband/sw/siw/siw_cm.c+2 -20 replies
10/14rdma/siw: create a temporary copy of private datadrivers/infiniband/sw/siw/siw_cm.c+16 -20 replies
11/14rdma/siw: use error and out logic at the end of siw_connect()drivers/infiniband/sw/siw/siw_cm.c+12 -70 replies
12/14rdma/siw: start mpa timer before calling siw_send_mpareqrep()drivers/infiniband/sw/siw/siw_cm.c+7 -50 replies
13/14rdma/siw: call the blocking kernel_bindconnect() just before siw_send_mpareqrep()drivers/infiniband/sw/siw/siw_cm.c+13 -120 replies
14/14rdma/siw: implement non-blocking connect.drivers/infiniband/sw/siw/siw_cm.c drivers/infiniband/sw/siw/siw_cm.h+96 -300 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 00/14] rdma/siw: implement non-blocking connect

Hi Bernard,

as written a few month ago, I have a patchset with a lot of fixes for siw.ko.

As requested I'm only send isolated chunks for easier review.

This is the first chunk adressing deadlocks in siw_connect()

The RDMA application layer expects rdma_connect() to be non-blocking as the completion is handled via RDMA_CM_EVENT_ESTABLISHED and other async events. It's not unlikely to hold a lock during the rdma_connect() call.

Without out this a connection attempt to a non-existing/reachable server block until the very long tcp timeout hits.
The application layer had no chance to have its own timeout handler as that would just deadlock with the already blocking rdma_connect().

First rdma_connect() holds id_priv->handler_mutex and deadlocks rdma_destroy_id().

And iw_cm_connect() called from within rdma_connect() sets IWCM_F_CONNECT_WAIT during the call to cm_id->device->ops.iw_connect(), siw_connect() in this case. It means that iw_cm_disconnect() and iw_destroy_cm_id() will both deadlock waiting for IWCM_F_CONNECT_WAIT being cleared.

Patch 1: Fixes a refcounting problem

Patches 2-7: Intruduces __siw_cep_terminate_upcall() making he upcall handling much more consistent handling more state combinations.

Patches 8-13 are preparation patches to siw_connect() in order to do the real non-blocking split in Patch 14.

Please have a look.

Thanks!
metze

Fixed issues in v2:
- Include more preparation patches related to __siw_cep_terminate_upcall() bases on review from Cheng Xu <[email protected]>

Stefan Metzmacher (14):
rdma/siw: remove superfluous siw_cep_put() from siw_connect() error
path
rdma/siw: make siw_cm_upcall() a noop without valid 'id' rdma/siw: split out a __siw_cep_terminate_upcall() function rdma/siw: use __siw_cep_terminate_upcall() for indirect
SIW_CM_WORK_CLOSE_LLP
rdma/siw: use __siw_cep_terminate_upcall() for SIW_CM_WORK_PEER_CLOSE rdma/siw: use __siw_cep_terminate_upcall() for SIW_CM_WORK_MPATIMEOUT rdma/siw: handle SIW_EPSTATE_CONNECTING in
__siw_cep_terminate_upcall()
rdma/siw: make use of kernel_{bind,connect,listen}() rdma/siw: let siw_connect() set AWAIT_MPAREP before
siw_send_mpareqrep()
rdma/siw: create a temporary copy of private data rdma/siw: use error and out logic at the end of siw_connect() rdma/siw: start mpa timer before calling siw_send_mpareqrep() rdma/siw: call the blocking kernel_bindconnect() just before
siw_send_mpareqrep()
rdma/siw: implement non-blocking connect.

 drivers/infiniband/sw/siw/siw_cm.c | 347 ++++++++++++++++++-----------
 drivers/infiniband/sw/siw/siw_cm.h |   1 +
 2 files changed, 224 insertions(+), 124 deletions(-)
Signature · 1 line
2.34.1