sr-iov and bridges (mlx4)
Subsystems (2)
Hello. I have a Mellanox ConnectX-3 Pro EN (MCX314A-BCCT). I'm only using a single port so it must provide IP for my host as well as connectivity for VMs. SR-IOV VFs are working great, my KVM VMs have Ethernet and RDMA.
However, I also want to support virtio VMs. Assuming eth0 is the first port on my mlx nic, I've tried placing VMs on a bridge with the primary physical interface, and giving an IP for management to a VF:
br0
|--- eth0 |--- VM |--- VM
vf0 (IP)
vf1 -> VM
vf2 -> VM
vf3 -> VM
I've tried placing VMs on a bridge with one of the VFs and using the primary iface for IP.
eth0 (IP)
br0
|--- vf0 |--- VM |--- VM
vf1 -> VM
vf2 -> VM
vf3 -> VM
And I've also tried using a veth pair to really spread things out:
br0 (IP)
|--- eth0
|--- veth-a
br1 |
|--- veth-b |--- VM |--- VM
vf1 -> VM
vf2 -> VM
vf3 -> VM
In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM.
Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...?
Thanks!
Matt
On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco
<[email protected]> wrote:
[...]
Quoted 3 lines
In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM.
[...]
Quoted 2 lines
Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...?
It would be a bit hard for bridge based promiscuous environment to work OOB for one of the functions (PF or VF) in SRIOV, this is generic issue, and not related
to specific vendor.
You need to use the bridge (8) tool (part of iproute2) and add the P.V VM MACs to
the PF interface as "self" see some slides (21/22/23) from netdev 0.1 that deal with that
https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf
Or.
It also possible to mark one bridge port (the PF) as non-promiscuous, but I haven't
played with that myself yet (slide 32), so can't just send you doing it...
Quoted 21 lines · Or Gerlitz
On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco
<[email protected]> wrote:
[...]Quoted 3 lines
In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM.
[...]
Quoted 2 lines
Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...?
It would be a bit hard for bridge based promiscuous environment to work OOB for one of the functions (PF or VF) in SRIOV, this is generic issue, and not related
to specific vendor.You need to use the bridge (8) tool (part of iproute2) and add the P.V VM MACs to
the PF interface as "self" see some slides (21/22/23) from netdev 0.1 that deal with that
https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf
Quoted 6 lines
Or.
It also possible to mark one bridge port (the PF) as non-promiscuous, but I haven't
played with that myself yet (slide 32), so can't just send you doing it...
Many thanks, that helps. Now to figure out the best approach for shoving this into OpenStack =)
Quoted 4 lines · Nieścierowicz Adam
Hi Matthew,
in near future i want to do exactly the same, if you make progress with SR-IOV+Bridge+OpenStack please leave here some advice.
I wrote a shell script which polls (~15s) each bridge /sys/class/net/brq*, and for each bridge figures out the uplink (not named /sys/class/net/brq*/brif/tap*). Then, for each port /sys/class/net/<bridge>/brif/tap*, determine the VM mac address and add an fdb entry if it doesn't already exist. In my environment, it seems the VM mac is the tap mac s/^fe:/fa:/.
This is a little messier than the non-promsicuous bridge option, but I wasn't able to get that working on CentOS7/kernel-ml-4.2/iproute-3.10. But either way, this won't work well if you're trying to do nested virt, which thankfully I don't need at this time.
If you're interested I can attach the script and systemd unit. Otherwise, I'll be looking to try to get this handled properly by neutron-linuxbridge-agent.
Hi Matthew,
in near future i want to do exactly the same, if you make progress with SR-IOV+Bridge+OpenStack please leave here some advice.
---
Pozdrawiam
Adam Nieścierowicz
W dniu 20.09.2015 18:21, Matthew Monaco napisał(a):
Quoted 17 lines
On 09/20/2015 05:01 AM, Or Gerlitz wrote: On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco <[email protected]> wrote:
[...] In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM. [...] Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...? It would be a bit hard for bridge based promiscuous environment to work OOB for one of the functions (PF or VF) in SRIOV, this is generic issue,
and not related
to specific vendor.You need to use the bridge (8) tool (part of iproute2) and add the P.V VM MACs to
the PF interface as "self" see some slides (21/22/23) from netdev 0.1 that deal with that
https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf [1]
Quoted 5 lines
Or.
It also possible to mark one bridge port (the PF) as non-promiscuous, but I haven't
played with that myself yet (slide 32), so can't just send you doing it...
Many thanks, that helps. Now to figure out the best approach for
shoving this
into OpenStack =)
Links:
------
[1] https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf