Skip to contentSkip to conversation

SR-IOV and KVM?

DISCUSSION Chris Friesen 8 participants 12 messages Jun 24, 2011 – Jun 28, 2011 KVM

lore
Share & tools
lore
#0 lore raw

Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?

Thanks,
Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#1 lore raw

I have tested the performance of sr-iov for kvm, it better than virtio-net.

2011/6/25 Chris Friesen <[email protected]>:

Quoted 19 lines

Is anyone working on SR-IOV for KVM, specifically for networking?  Or is virtio good enough that it's not a pressing concern?

Thanks,
Chris

--
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to Chris Friesen
Quoted 3 lines · Chris Friesen

Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?

What exactly are you missing now?

Quoted 4 lines

Thanks,
Chris

Signature · 1 line
Sasha.
#3 lore raw
Quoted 6 lines · Sasha Levin
Quoted 3 lines · Chris Friesen

Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?

What exactly are you missing now?

Maybe it's just a documentation thing. I'm just getting into KVM and trying to figure everything out, and I don't see a step-by-step guide to configuring a host/guest system with either SR-IOV or Intel's VMDq.

For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?

Do I need to configure a software bridge to allow the guests to talk to each other? I assume the passthrough require VT-d support in the hardware and host kernel--is that correct?

Thanks,
Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#4 lore raw
Quoted 2 lines · Chris Friesen

Maybe it's just a documentation thing. I'm just getting into KVM and trying to figure everything out, and I don't see a step-by-step guide to configuring a host/guest system with either SR-IOV or Intel's VMDq.

You can use virt-manager; it has a GUI for that. Once you're comfortable with it you can drop down to virsh command line and from there to raw qemu command line / monitor.

Quoted 2 lines

For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?

I think all that is needed is to prevent the host from binding to the guest - not sure.

Quoted 1 line

Do I need to configure a software bridge to allow the guests to talk to each other?

S/R-IOV networking is beyond my knowledge - I think it depends on your switch, sometimes it will loop back packets to other guests, other times the NIC itself will do that. It may just work, so try it out.

Quoted 2 lines

I assume the passthrough require VT-d support in the hardware and host kernel--is that correct?

Yes. Recent kernels have that.

Signature · 1 line
error compiling committee.c: too many arguments to function
#5 lore raw
Quoted 9 lines · Avi Kivity
Quoted 4 lines · Chris Friesen

Maybe it's just a documentation thing. I'm just getting into KVM and trying to figure everything out, and I don't see a step-by-step guide to configuring a host/guest system with either SR-IOV or Intel's VMDq.

You can use virt-manager; it has a GUI for that. Once you're comfortable with it you can drop down to virsh command line and from there to raw qemu command line / monitor.

To some virt-manager is more painful than the command line is not too painful. I have a set of home grown scripts. This is the code I added for PCI devices:

function claim_pci_dev

{
        local pciid=$1
        local domain=0000   # TO-DO: determine this automatically
        local id

        [ -z "$pciid" ] && return 1

        local drv=$(lspci -s ${pciid} -k | awk '$0 ~ /Kernel driver in
use:/ {print $NF}')
        [ "$drv" = "pci-stub" ] && return 0

        id=$(lspci -n | awk  '{if ($1 == "'${pciid}'") {print $3}}')
        id=${id/:/ }
        if [ -z "$id" ]
        then
                err "failed to find vendor-product id for PCI id \"$pciid\""
                return 1
        fi

        echo "$id" > /sys/bus/pci/drivers/pci-stub/new_id
        if [ $? -ne 0 ]
        then
                err "Failed to tell pci-stub about id \"$id\""
                return 1
        fi

        echo "${domain}:${pciid}" >
/sys/bus/pci/devices/${domain}:${pciid}/driver/unbind
        if [ $? -ne 0 ]
        then
                err "Failed to unbind PCI device \"${domain}:${pciid}\""
                return 1
        fi

        # pci-stub claims device so it can be assigned guest
        echo "${domain}:${pciid}" > /sys/bus/pci/drivers/pci-stub/bind
        if [ $? -ne 0 ]
        then
                err "Failed to bind pci-stub to device

\"${domain}:${pciid}\""
return 1
fi

return 0
}

e.g., claim_pci_dev "06:10.0"

PF-VF correlations. Example using eth2
INTFC=eth2

# pci id to VF correlation:
DEVPATH=$(readlink -f /sys/class/net/${INTFC})
ls -l ${DEVPATH}/device/virtfn*
# map a virtual function to a netdevice - in this case find the PCI id
for VF 0
VFN=0
VFID=$(readlink  ${DEVPATH}/device/virtfn${VFN} | sed -e 's,^../,,')

--> VFID is the PCI device ID for VF 0 on netdevice eth2

qemu-kvm command line addition is:
-device pci-assign,host=${PCIID}

Quoted 8 lines
Quoted 5 lines

For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?

I think all that is needed is to prevent the host from binding to the guest - not sure.

See above.

Quoted 7 lines
Quoted 3 lines

Do I need to configure a software bridge to allow the guests to talk to each other?

S/R-IOV networking is beyond my knowledge - I think it depends on your switch, sometimes it will loop back packets to other guests, other times the NIC itself will do that. It may just work, so try it out.

PF for the NIC is attached to the network. VFs within the nic are attached by extension. VF-VF packet flows (ie., VM to VM on the same host) work more efficiently than hitting the wire, but not as well as virtio connected to a bridge:
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

Very good descriptions in the companion guide at
http://www.intel.com/products/ethernet/resource.htm#s1=Gigabit%20Ethernet&s2=82576EB&s3=all

David

Quoted 7 lines
Quoted 4 lines

I assume the passthrough require VT-d support in the hardware and host kernel--is that correct?

Yes. Recent kernels have that.

#6 lore raw
In reply to Chris Friesen

Chris Friesen <[email protected]> wrote on 06/27/2011 09:15:05 AM:

Quoted 2 lines
Quoted 2 lines · Chris Friesen
Quoted 1 line

Is anyone working on SR-IOV for KVM, specifically for networking? Or

is

Quoted 1 line
Quoted 1 line
Quoted 1 line

virtio good enough that it's not a pressing concern?

In IBM's recent postings of results for the SPECvirt benchmark on RHEL 6.0
[1] we used SR-IOV for the network adapter for most of the guests. The adapters in the host were the Intel x520 10 Gigabit Server adapters (lscpi says Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev
22) ).

virtio may be good enough for lighter workloads, but there was no way we could get the higher guest counts for SPECvirt with the performance of virtio. We needed the performance of SR-IOV.

Quoted 3 lines

For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?

That is correct. You also need to make sure your guest has the driver for the virtual function.

As an example, on my qemu command line I have the options:
-pcidevice host=8b:10.0\
-net none\

I use the "-net none" option because the qemu I am running likes to do you a favor and create a network device for you if you didn't specify one (it doesn't know that the PCI device is a network device). Without "-net none", what can happen is that qemu will create an emulated network device for you (e.g., e1000), and the guest will most likely have a driver for it that it will load. If you don't have the driver in the guest for the virtual function, you can still get network connectivity to the guest and may think you are using SR-IOV when you are not.

Quoted 2 lines

Do I need to configure a software bridge to allow the guests to talk to each other?

You do not need to configure a software bridge. The adapter on the host behaves like a switch and routes traffic between the virtual functions. (I don't know offhand if the traffic between the virtual functions is looped back by the adapter or if it actually goes out the adapter port and comes back in.)

Steve D.

[1] http://www.spec.org/virt_sc2010/results/specvirt_sc2010_perf.html

#8 lore raw
Quoted 6 lines · Troels Arvin

Hello,

Quoted 1 line · Chris Friesen

Is anyone working on SR-IOV for KVM, specifically for networking?

KVM is one of the few virtualization systems which supports SR-IOV.

82576EB and a Dell R410 are working fine for me, but my host OS is Fedora 14, not RHEL6.
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

David

Quoted 5 lines

But my experience is that it's hard to find a hardware combination which really lets you use it:
http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/40981

#9 lore raw
Quoted 2 lines · David Ahern

82576EB and a Dell R410 are working fine for me, but my host OS is Fedora 14, not RHEL6.

82576EB is a dual-port embedded NIC?

(The 82576 which is in "my" R710 is a quad-port PCIe card.)

Does your setup allow for SR-IOV without having to turn off ACS? (ACS is a security feature.)

Quoted 1 line

http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

Interesting thread, thanks.

Signature · 3 lines
Regards,
Troels Arvin <[email protected]>
http://troels.arvin.dk/
#10 lore raw
Quoted 5 lines · Troels Arvin
Quoted 2 lines · David Ahern

82576EB and a Dell R410 are working fine for me, but my host OS is Fedora 14, not RHEL6.

82576EB is a dual-port embedded NIC?

I have a quad-port PCIe card.

Quoted 5 lines

(The 82576 which is in "my" R710 is a quad-port PCIe card.)

Does your setup allow for SR-IOV without having to turn off ACS? (ACS is a security feature.)

I recall having to twiddle some BIOS settings - and doing so in iterations. I forget which settings exactly. I'll see if Dell has a tool to dump BIOS settings to a file that works with Fedora 14.

David

Quoted 6 lines
Quoted 3 lines

http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

Interesting thread, thanks.

#11 lore raw
Quoted 4 lines · David Ahern
Quoted 2 lines

Does your setup allow for SR-IOV without having to turn off ACS? (ACS is a security feature.)

I recall having to twiddle some BIOS settings

I believe that if you turn off ACS, it's through a kernel module option.

Signature · 3 lines
Regards,
Troels Arvin <[email protected]>
http://troels.arvin.dk/