RoCE passive side failures on 4.4-rc5

Guys,

Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today, I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong works).

I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the one with the 4.4-rc5 kernel can act as both (rping) client/server for IB links but only (rping) client for RoCE.

I tried both inter-node and loopback runs, in all cases, the client side getsCM reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, TCP) works OK.

Or.

[1] Linus tree up to commit a5e90b1 "Merge branch 'fixes' of git://ftp.arm.linux.org.uk/~rmk/linux-arm"

[2] $ rping -d -v -c -a 192.168.31.17 -C 1
verbose
client
count 1
created cm_id 0x60cd90
cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x60cd90 (parent) cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x60cd90 (parent) rdma_resolve_addr - rdma_resolve_route successful
created pd 0x60cfe0
created channel 0x608510
created cq 0x607f50
created qp 0x60d000
rping_setup_buffers called on cb 0x606010
allocated & registered buffers...
cq_thread started.
wait for CONNECTED state 10
cma_event type RDMA_CM_EVENT_REJECTED cma_id 0x60cd90 (parent) connect error -1
rping_free_buffers called on cb 0x606010
cma event RDMA_CM_EVENT_REJECTED, error 28

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I'm using 4.4-rc2+ and I have RoCE working.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I tried 4.4-rc2 and I see the same problem. I will make my .config available to you and Moni so you can try it out on your systems Sunday.

I see this over both CX2 and CX3-pro, both are in VPI config.

BTW getting the 28 (consumer defined) reject reason is something I saw coming and going from time to time in various weird cases over the past years.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

OK, small progress, when the force Eth link type on my IB port (using mlx4 sysfs), things work.

You should be able to reproduce it on your non-VPI systems the other way around, by forcing IB link type on one of the Eth ports and see the failure.

I Saw the same behavior with both 4.4-rc2 and 4.4-rc5

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I've posted a patch that fixes that, please take a look at [1].

Regards,
Matan

[1] https://www.mail-archive.com/[email protected]/msg30777.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I've been seeing this too in my 4.4-rc testing, so I'll have test results today.

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD