Skip to contentSkip to conversation

rxe mcast

DISCUSSION Bob Pearson 2 participants 3 messages Nov 1, 2023 InfiniBand

lore
Share & tools
lore
#0 lore raw

Jason,

I wrote a test program to verify that multicast is working in rxe since there don't seem to be any pyverbs tests that actually test this. It turns out that it doesn't work for several reasons most of which I have fixed. But there is one that I can't figure out.

The rxe driver calls dev_mc_add() with a MAC address to add the multicast MAC address to the device which shows up when you type 'ip maddr'. But nothing comes through from the network to the driver. Wireshark does see the packets but they don't get to the IP or UDP layers in the netdev stack.

If I run iperf -u and bind to a multicast address it works fine and
when you type ip maddr you get both a link: MAC address and an
inet: IP multicast address. In fact if you start an iperf server with the same multicast address I am testing with and just let it sit there my test traffic also come through to the rxe driver.
So creating the IP mcast address seems critical to letting the
netdev stack receive traffic.

I tried creating a separate UDP socket bound to the mcast address but it doesn't seem to create the required IP address.

Any ideas would be welcome.

Bob

maintainer
#1 lore raw
Quoted 7 lines · Bob Pearson

Jason,

I wrote a test program to verify that multicast is working in rxe since there don't seem to be any pyverbs tests that actually test this. It turns out that it doesn't work for several reasons most of which I have fixed. But there is one that I can't figure out.

Quoted 5 lines

The rxe driver calls dev_mc_add() with a MAC address to add the multicast MAC address to the device which shows up when you type 'ip maddr'. But nothing comes through from the network to the driver. Wireshark does see the packets but they don't get to the IP or UDP layers in the netdev stack.

I think you also need to attach a multicast IP address to the RXE socket ?

Quoted 2 lines

So creating the IP mcast address seems critical to letting the netdev stack receive traffic.

Yes

Quoted 2 lines

I tried creating a separate UDP socket bound to the mcast address but it doesn't seem to create the required IP address.

I don't think that is how you do it... There is a set sock opt thing you need, IIRC. It has been a long time since I last wrote code for it

Check what iperf did and that will be the basic thing that has to be cloned in the kernel

Jason

#2 lore raw
Quoted 25 lines · Jason Gunthorpe
Quoted 7 lines · Bob Pearson

Jason,

I wrote a test program to verify that multicast is working in rxe since there don't seem to be any pyverbs tests that actually test this. It turns out that it doesn't work for several reasons most of which I have fixed. But there is one that I can't figure out.

Quoted 5 lines

The rxe driver calls dev_mc_add() with a MAC address to add the multicast MAC address to the device which shows up when you type 'ip maddr'. But nothing comes through from the network to the driver. Wireshark does see the packets but they don't get to the IP or UDP layers in the netdev stack.

I think you also need to attach a multicast IP address to the RXE socket ?

Quoted 2 lines

So creating the IP mcast address seems critical to letting the netdev stack receive traffic.

Yes

Quoted 2 lines

I tried creating a separate UDP socket bound to the mcast address but it doesn't seem to create the required IP address.

Quoted 7 lines

I don't think that is how you do it... There is a set sock opt thing you need, IIRC. It has been a long time since I last wrote code for it

Check what iperf did and that will be the basic thing that has to be cloned in the kernel

Jason

Thanks Jason,

That found it. I traced set socket opt down to ip_mc_join/leave_group and ipv6_sock_mc_join/drop which seems to be working. Now ip maddr reports both the link and inet addresses and packets are getting up to the rxe driver without running iperf on the side.

Some fixes coming soon.

Bob