slow rdma_connect

Hi,

We have a client-sever application that use rdma cm to establish connections
to do RDMA. It takes around 30 or 40 secs for 500 clients started simultaneously
to complete the connections to a server. The machine where we are testing has enough CPU resource , so this 30 - 40secs appears to be the time for rdma_cm to transition from resolving address to establishing connections.. Is this expected or should we be configuring rdma cm to handle this load ?

set of system calls involved in both sides are:

server:
rdma_create_id, rdma_listen, loop with { select, rdma_get_cm_event} (listen backlog is 10)

client:
rdma_create_id, rdma_resolve_addr, rdma_resolve_route, rdma_connect, loop with { select rdma_get_cm_event} (resolve timeouts are 20ms,
connect parameters are :
param.responder_resources = 1;

  param.initiator_depth     = 1;
  param.rnr_retry_count     = 7;
  param.retry_count         = 7;

}

versions:
libibcm-1.0.5-1
ibacm-1.0.3-1
librdmacm-1.0.13-1

--
thanks,
arun

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The rdma_cm can handle this load, but...

the backlog is the issue. With a backlog of only 10, all additional incoming requests get dropped and must be retried. The connect retry is a couple of seconds. Try increasing the backlog (or setting it to 0).

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html