rsockets with RoCE
Hi Sean,
I attempted to use rsockets with ConnectX-EN adapters and the client receives a "Connection refused" message. I debugged this a bit further and see that the client is actually receiving an IB_CM_REJ_RECEIVED with reason being 28 i.e. "Consumer Reject". Could this be because of the difference in how GIDs are obtained in the case of RoCE? Have you attempted rsockets on RoCE at all?
Incidentally this problem also highlights another issue we have encountered previously. The only way to debug issues like this in RDMA CM is to modify the kernel module. Currently there is no way we can turn on (and off) debug and get insights into the connection set up process. Are you open to considering patches to enhance debug with RDMA CM, particularly during the address resolution and connection setup process. That way it should not impact the performance during data transfer.
Thanks
Pradeep
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4FEB964B.5060303-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org
Quoted 6 lines
I attempted to use rsockets with ConnectX-EN adapters and the client receives a "Connection refused" message. I debugged this a bit further and see that the client is actually receiving an
IB_CM_REJ_RECEIVEDwith reason being 28 i.e. "Consumer Reject". Could this be because of the difference in how GIDs are obtained in the case of RoCE? Have you attempted rsockets on RoCE at all?
I have not tested this on RoCE. Do other librdmacm examples (e.g. ucmatose or rdma_client/server) work in your configuration?
Quoted 7 lines
Incidentally this problem also highlights another issue we have encountered previously. The only way to debug issues like this in RDMA CM is to modify the kernel module. Currently there is no way we can turn on (and off) debug and get insights into the connection set up process. Are you open to considering patches to enhance debug with RDMA CM, particularly during the address resolution and connection setup process. That way it should not impact the performance during data transfer.
No objection. The rdma_cm shouldn't be considered speed path anyway. Btw, the IB CM exports some counters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237346A3F493-P5GAC/[email protected]
Quoted 7 lines · Sean
Quoted 6 lines
I attempted to use rsockets with ConnectX-EN adapters and the client receives a "Connection refused" message. I debugged this a bit further and see that the client is actually receiving an
IB_CM_REJ_RECEIVEDwith reason being 28 i.e. "Consumer Reject". Could this be because of the difference in how GIDs are obtained in the case of RoCE? Have you attempted rsockets on RoCE at all?I have not tested this on RoCE. Do other librdmacm examples (e.g. ucmatose or rdma_client/server) work in your configuration?
I have attempted running the utilities in the rsocket/examples/ directory like ucmatose and rping. ucamtose fails identically:
cmatose: event: RDMA_CM_EVENT_REJECTED, error: 28
and rping fails as follows:
rdma_resolve_route: No such device
waiting for addr/route resolution state 10
Quoted 9 lines
Quoted 8 lines
Incidentally this problem also highlights another issue we have encountered previously. The only way to debug issues like this in RDMA CM is to modify the kernel module. Currently there is no way we can turn on (and off) debug and get insights into the connection set up process. Are you open to considering patches to enhance debug with RDMA CM, particularly during the address resolution and connection setup process. That way it should not impact the performance during data transfer.
No objection. The
rdma_cmshouldn't be considered speed path anyway. Btw, the IB CM exports some counters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.
I have not used this before. How does one read these counters?
Thanks
Pradeep
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4FEBA31A.4050302-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org
Quoted 5 lines
Quoted 1 line
No objection. The
rdma_cmshouldn't be considered speed path anyway. Btw, the IB CM exports somecounters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.
I have not used this before. How does one read these counters?
Look in /sys/class/infiniband_cm, under the device/port subdirectories --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html