Skip to contentSkip to conversation

RDMA with IPv6 between 2 NICS on the same host is not working

DISCUSSION Andrei S. 1 participant 2 messages Jan 30, 2025 – Feb 3, 2025 4 messages not in the archive InfiniBand

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andrei S.

author
#4 lore raw

Hi,

I've noticed that RDMA doesn't work on IPv6 between 2 NIC on the same host.
I've tested with rping and ibv_rc_pingpong. Below you can find the commands and the behavior for each app.

rping test:

Quoted 2 lines

rping -s -a 2001:db8:153:1::10 -p 60009 -V -d // waits for a connection
rping -c -a 2001:db8:153:1::10 -I 2001:db8:171:2::20 -p 60009 -V -d // client waits indefinitely to connect to server

ibv_rc_pingpong test:

Quoted 2 lines

ibv_rc_pingpong -d mlx5_2 -g 5 // waits for a connection ibv_rc_pingpong -d mlx5_3 -g 5 <IPv4 address>  //client throws timeout error:

local address:  LID 0x0000, QPN 0x000150, PSN 0x23c9c7, GID 2001:db8:171:2::20 remote address: LID 0x0000, QPN 0x00012e, PSN 0x96791a, GID 2001:db8:153:1::10 Failed status transport retry counter exceeded (12) for wr_id 2 parse WC failed 1

What I have noticed with tcpdump on mlx5_3 is that the packets have Source MAC equal to Destination MAC, which is wrong! Also, I want to mention that both interfaces have also IPv4 addresses and if I'm using IPv4 everything works perfect!

Below you can find my setup:

Ubuntu 22.0.4.5 (6.8.0-49-generic #49~22.04.1-Ubuntu) with 2 NICs connected via a switch.

Network interfaces:

enp153s0np0:
...

inet6 fe80::a288:c2ff:fe7d:a8ca  prefixlen 64  scopeid 0x20<link> inet6 2001:db8:153:1::10  prefixlen 64  scopeid 0x0<global> ether a0:88:c2:7d:a8:ca  txqueuelen 1000  (Ethernet)

and

enp171s0np0:
...
inet6 fe80::a288:c2ff:fe7d:a88a  prefixlen 64  scopeid 0x20<link> inet6 2001:db8:171:2::20  prefixlen 64  scopeid 0x0<global> ether a0:88:c2:7d:a8:8a txqueuelen 1000  (Ethernet)

Here is the output of the ibv_devinfo:

hca_id: mlx5_2
        transport:                      InfiniBand (0) fw_ver:                         28.39.1002 node_guid:                      a088:c203:007d:a8ca sys_image_guid:                 a088:c203:007d:a8ca vendor_id:                      0x02c9 vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838 phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4) max_mtu:                4096 (5) active_mtu:             1024 (3) sm_lid:                 0 port_lid:               0 port_lmc:               0x00 link_layer:             Ethernet

hca_id: mlx5_3
        transport:                      InfiniBand (0) fw_ver:                         28.39.1002 node_guid:                      a088:c203:007d:a88a sys_image_guid:                 a088:c203:007d:a88a vendor_id:                      0x02c9 vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838 phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4) max_mtu:                4096 (5) active_mtu:             1024 (3) sm_lid:                 0 port_lid:               0 port_lmc:               0x00 link_layer:             Ethernet

Thank you,
Andrei

author
#5 lore raw

RDMA over IPv6 between 2 NICS on the same host is not working

Hello,

I've resent the mail with correct subject.

I've noticed that RDMA doesn't work on IPv6 between 2 NIC on the same host.
I've tested with rping and ibv_rc_pingpong. Below you can find the commands and the behavior for each app.

rping test:

Quoted 2 lines

rping -s -a 2001:db8:153:1::10 -p 60009 -V -d // waits for a connection
rping -c -a 2001:db8:153:1::10 -I 2001:db8:171:2::20 -p 60009 -V -d // client waits indefinitely to connect to server

ibv_rc_pingpong test:

Quoted 2 lines

ibv_rc_pingpong -d mlx5_2 -g 5 // waits for a connection ibv_rc_pingpong -d mlx5_3 -g 5 <IPv4 address>  //client throws timeout error:

local address:  LID 0x0000, QPN 0x000150, PSN 0x23c9c7, GID 2001:db8:171:2::20 remote address: LID 0x0000, QPN 0x00012e, PSN 0x96791a, GID 2001:db8:153:1::10 Failed status transport retry counter exceeded (12) for wr_id 2 parse WC failed 1

What I have noticed with tcpdump on mlx5_3 is that the packets have Source MAC equal to Destination MAC, which is wrong! Also, I want to mention that both interfaces have also IPv4 addresses and if I'm using IPv4 everything works perfect!

Below you can find my setup:

Ubuntu 22.0.4.5 (6.8.0-49-generic #49~22.04.1-Ubuntu) with 2 NICs connected via a switch.

Network interfaces:

enp153s0np0:
...

inet6 fe80::a288:c2ff:fe7d:a8ca  prefixlen 64  scopeid 0x20<link> inet6 2001:db8:153:1::10  prefixlen 64  scopeid 0x0<global> ether a0:88:c2:7d:a8:ca  txqueuelen 1000  (Ethernet)

and

enp171s0np0:
...
inet6 fe80::a288:c2ff:fe7d:a88a  prefixlen 64  scopeid 0x20<link> inet6 2001:db8:171:2::20  prefixlen 64  scopeid 0x0<global> ether a0:88:c2:7d:a8:8a txqueuelen 1000  (Ethernet)

Here is the output of the ibv_devinfo:

hca_id: mlx5_2
        transport:                      InfiniBand (0) fw_ver:                         28.39.1002 node_guid:                      a088:c203:007d:a8ca sys_image_guid:                 a088:c203:007d:a8ca vendor_id:                      0x02c9 vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838 phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4) max_mtu:                4096 (5) active_mtu:             1024 (3) sm_lid:                 0 port_lid:               0 port_lmc:               0x00 link_layer:             Ethernet

hca_id: mlx5_3
        transport:                      InfiniBand (0) fw_ver:                         28.39.1002 node_guid:                      a088:c203:007d:a88a sys_image_guid:                 a088:c203:007d:a88a vendor_id:                      0x02c9 vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838 phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4) max_mtu:                4096 (5) active_mtu:             1024 (3) sm_lid:                 0 port_lid:               0 port_lmc:               0x00 link_layer:             Ethernet

Thank you,
Andrei