RDMA with IPv6 between 2 NICS on the same host is not working
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
I've noticed that RDMA doesn't work on IPv6 between 2 NIC on the same host.
I've tested with rping and ibv_rc_pingpong. Below you can find the commands and the behavior for each app.
rping test:
Quoted 2 lines
rping -s -a 2001:db8:153:1::10 -p 60009 -V -d // waits for a connection
rping -c -a 2001:db8:153:1::10 -I 2001:db8:171:2::20 -p 60009 -V -d // client waits indefinitely to connect to server
ibv_rc_pingpong test:
Quoted 2 lines
ibv_rc_pingpong-dmlx5_2-g 5 // waits for a connectionibv_rc_pingpong-dmlx5_3-g 5 <IPv4 address> //client throws timeout error:
local address: LID 0x0000, QPN 0x000150, PSN 0x23c9c7, GID 2001:db8:171:2::20 remote address: LID 0x0000, QPN 0x00012e, PSN 0x96791a, GID 2001:db8:153:1::10 Failed status transport retry counter exceeded (12) for wr_id 2 parse WC failed 1
What I have noticed with tcpdump on mlx5_3 is that the packets have Source MAC equal to Destination MAC, which is wrong! Also, I want to mention that both interfaces have also IPv4 addresses and if I'm using IPv4 everything works perfect!
Below you can find my setup:
Ubuntu 22.0.4.5 (6.8.0-49-generic #49~22.04.1-Ubuntu) with 2 NICs connected via a switch.
Network interfaces:
enp153s0np0:
...
inet6 fe80::a288:c2ff:fe7d:a8ca prefixlen 64 scopeid 0x20<link> inet6 2001:db8:153:1::10 prefixlen 64 scopeid 0x0<global> ether a0:88:c2:7d:a8:ca txqueuelen 1000 (Ethernet)
and
enp171s0np0:
...
inet6 fe80::a288:c2ff:fe7d:a88a prefixlen 64 scopeid 0x20<link> inet6 2001:db8:171:2::20 prefixlen 64 scopeid 0x0<global> ether a0:88:c2:7d:a8:8a txqueuelen 1000 (Ethernet)
Here is the output of the ibv_devinfo:
hca_id: mlx5_2
transport: InfiniBand (0) fw_ver: 28.39.1002 node_guid: a088:c203:007d:a8ca sys_image_guid: a088:c203:007d:a8ca vendor_id: 0x02c9 vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838 phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
hca_id: mlx5_3
transport: InfiniBand (0) fw_ver: 28.39.1002 node_guid: a088:c203:007d:a88a sys_image_guid: a088:c203:007d:a88a vendor_id: 0x02c9 vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838 phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
Thank you,
Andrei
RDMA over IPv6 between 2 NICS on the same host is not working
Hello,
I've resent the mail with correct subject.
I've noticed that RDMA doesn't work on IPv6 between 2 NIC on the same host.
I've tested with rping and ibv_rc_pingpong. Below you can find the commands and the behavior for each app.
rping test:
Quoted 2 lines
rping -s -a 2001:db8:153:1::10 -p 60009 -V -d // waits for a connection
rping -c -a 2001:db8:153:1::10 -I 2001:db8:171:2::20 -p 60009 -V -d // client waits indefinitely to connect to server
ibv_rc_pingpong test:
Quoted 2 lines
ibv_rc_pingpong-dmlx5_2-g 5 // waits for a connectionibv_rc_pingpong-dmlx5_3-g 5 <IPv4 address> //client throws timeout error:
local address: LID 0x0000, QPN 0x000150, PSN 0x23c9c7, GID 2001:db8:171:2::20 remote address: LID 0x0000, QPN 0x00012e, PSN 0x96791a, GID 2001:db8:153:1::10 Failed status transport retry counter exceeded (12) for wr_id 2 parse WC failed 1
What I have noticed with tcpdump on mlx5_3 is that the packets have Source MAC equal to Destination MAC, which is wrong! Also, I want to mention that both interfaces have also IPv4 addresses and if I'm using IPv4 everything works perfect!
Below you can find my setup:
Ubuntu 22.0.4.5 (6.8.0-49-generic #49~22.04.1-Ubuntu) with 2 NICs connected via a switch.
Network interfaces:
enp153s0np0:
...
inet6 fe80::a288:c2ff:fe7d:a8ca prefixlen 64 scopeid 0x20<link> inet6 2001:db8:153:1::10 prefixlen 64 scopeid 0x0<global> ether a0:88:c2:7d:a8:ca txqueuelen 1000 (Ethernet)
and
enp171s0np0:
...
inet6 fe80::a288:c2ff:fe7d:a88a prefixlen 64 scopeid 0x20<link> inet6 2001:db8:171:2::20 prefixlen 64 scopeid 0x0<global> ether a0:88:c2:7d:a8:8a txqueuelen 1000 (Ethernet)
Here is the output of the ibv_devinfo:
hca_id: mlx5_2
transport: InfiniBand (0) fw_ver: 28.39.1002 node_guid: a088:c203:007d:a8ca sys_image_guid: a088:c203:007d:a8ca vendor_id: 0x02c9 vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838 phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
hca_id: mlx5_3
transport: InfiniBand (0) fw_ver: 28.39.1002 node_guid: a088:c203:007d:a88a sys_image_guid: a088:c203:007d:a88a vendor_id: 0x02c9 vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838 phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
Thank you,
Andrei