SR-IOV and KVM?
Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?
Thanks,
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
I have tested the performance of sr-iov for kvm, it better than virtio-net.
2011/6/25 Chris Friesen <[email protected]>:
Quoted 19 lines
Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?
Thanks,
Chris--
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Chris Friesen
Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?
What exactly are you missing now?
Quoted 4 lines
Thanks,
Chris
Signature · 1 line
Sasha.
Quoted 6 lines · Sasha Levin
Quoted 3 lines · Chris Friesen
Is anyone working on SR-IOV for KVM, specifically for networking? Or is virtio good enough that it's not a pressing concern?
What exactly are you missing now?
Maybe it's just a documentation thing. I'm just getting into KVM and trying to figure everything out, and I don't see a step-by-step guide to configuring a host/guest system with either SR-IOV or Intel's VMDq.
For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?
Do I need to configure a software bridge to allow the guests to talk to each other? I assume the passthrough require VT-d support in the hardware and host kernel--is that correct?
Thanks,
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
Quoted 2 lines · Chris Friesen
Maybe it's just a documentation thing. I'm just getting into KVM and trying to figure everything out, and I don't see a step-by-step guide to configuring a host/guest system with either SR-IOV or Intel's VMDq.
You can use virt-manager; it has a GUI for that. Once you're comfortable with it you can drop down to virsh command line and from there to raw qemu command line / monitor.
Quoted 2 lines
For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?
I think all that is needed is to prevent the host from binding to the guest - not sure.
Quoted 1 line
Do I need to configure a software bridge to allow the guests to talk to each other?
S/R-IOV networking is beyond my knowledge - I think it depends on your switch, sometimes it will loop back packets to other guests, other times the NIC itself will do that. It may just work, so try it out.
Quoted 2 lines
I assume the passthrough require VT-d support in the hardware and host kernel--is that correct?
Yes. Recent kernels have that.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 4 lines · Chris Friesen
Maybe it's just a documentation thing. I'm just getting into KVM and trying to figure everything out, and I don't see a step-by-step guide to configuring a host/guest system with either SR-IOV or Intel's VMDq.
You can use virt-manager; it has a GUI for that. Once you're comfortable with it you can drop down to virsh command line and from there to raw qemu command line / monitor.
To some virt-manager is more painful than the command line is not too painful. I have a set of home grown scripts. This is the code I added for PCI devices:
function claim_pci_dev
{
local pciid=$1
local domain=0000 # TO-DO: determine this automatically
local id
[ -z "$pciid" ] && return 1
local drv=$(lspci -s ${pciid} -k | awk '$0 ~ /Kernel driver in
use:/ {print $NF}')
[ "$drv" = "pci-stub" ] && return 0
id=$(lspci -n | awk '{if ($1 == "'${pciid}'") {print $3}}')
id=${id/:/ }
if [ -z "$id" ]
then
err "failed to find vendor-product id for PCI id \"$pciid\""
return 1
fi
echo "$id" > /sys/bus/pci/drivers/pci-stub/new_id
if [ $? -ne 0 ]
then
err "Failed to tell pci-stub about id \"$id\""
return 1
fi
echo "${domain}:${pciid}" >
/sys/bus/pci/devices/${domain}:${pciid}/driver/unbind
if [ $? -ne 0 ]
then
err "Failed to unbind PCI device \"${domain}:${pciid}\""
return 1
fi
# pci-stub claims device so it can be assigned guest
echo "${domain}:${pciid}" > /sys/bus/pci/drivers/pci-stub/bind
if [ $? -ne 0 ]
then
err "Failed to bind pci-stub to device\"${domain}:${pciid}\""
return 1
fi
return 0
}
e.g., claim_pci_dev "06:10.0"
PF-VF correlations. Example using eth2
INTFC=eth2
# pci id to VF correlation:
DEVPATH=$(readlink -f /sys/class/net/${INTFC})
ls -l ${DEVPATH}/device/virtfn*# map a virtual function to a netdevice - in this case find the PCI id
for VF 0
VFN=0
VFID=$(readlink ${DEVPATH}/device/virtfn${VFN} | sed -e 's,^../,,')--> VFID is the PCI device ID for VF 0 on netdevice eth2
qemu-kvm command line addition is:
-device pci-assign,host=${PCIID}
Quoted 8 lines
Quoted 5 lines
For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?
I think all that is needed is to prevent the host from binding to the guest - not sure.
See above.
Quoted 7 lines
Quoted 3 lines
Do I need to configure a software bridge to allow the guests to talk to each other?
S/R-IOV networking is beyond my knowledge - I think it depends on your switch, sometimes it will loop back packets to other guests, other times the NIC itself will do that. It may just work, so try it out.
PF for the NIC is attached to the network. VFs within the nic are attached by extension. VF-VF packet flows (ie., VM to VM on the same host) work more efficiently than hitting the wire, but not as well as virtio connected to a bridge:
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686
Very good descriptions in the companion guide at
http://www.intel.com/products/ethernet/resource.htm#s1=Gigabit%20Ethernet&s2=82576EB&s3=all
David
Quoted 7 lines
Quoted 4 lines
I assume the passthrough require VT-d support in the hardware and host kernel--is that correct?
Yes. Recent kernels have that.
Chris Friesen <[email protected]> wrote on 06/27/2011 09:15:05 AM:
Quoted 2 lines
Quoted 2 lines · Chris Friesen
Quoted 1 line
Is anyone working on SR-IOV for KVM, specifically for networking? Or
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
virtio good enough that it's not a pressing concern?
In IBM's recent postings of results for the SPECvirt benchmark on RHEL 6.0
[1] we used SR-IOV for the network adapter for most of the guests. The adapters in the host were the Intel x520 10 Gigabit Server adapters (lscpi says Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev
22) ).
virtio may be good enough for lighter workloads, but there was no way we could get the higher guest counts for SPECvirt with the performance of virtio. We needed the performance of SR-IOV.
Quoted 3 lines
For SR-IOV I assume I need to configure the PF in the host, enable some number of VFs, pass through a VF to each guest and load the vf driver. Is that right?
That is correct. You also need to make sure your guest has the driver for the virtual function.
As an example, on my qemu command line I have the options:
-pcidevice host=8b:10.0\
-net none\
I use the "-net none" option because the qemu I am running likes to do you a favor and create a network device for you if you didn't specify one (it doesn't know that the PCI device is a network device). Without "-net none", what can happen is that qemu will create an emulated network device for you (e.g., e1000), and the guest will most likely have a driver for it that it will load. If you don't have the driver in the guest for the virtual function, you can still get network connectivity to the guest and may think you are using SR-IOV when you are not.
Quoted 2 lines
Do I need to configure a software bridge to allow the guests to talk to each other?
You do not need to configure a software bridge. The adapter on the host behaves like a switch and routes traffic between the virtual functions. (I don't know offhand if the traffic between the virtual functions is looped back by the adapter or if it actually goes out the adapter port and comes back in.)
Steve D.
[1] http://www.spec.org/virt_sc2010/results/specvirt_sc2010_perf.html
Hello,
Quoted 1 line · Chris Friesen
Is anyone working on SR-IOV for KVM, specifically for networking?
KVM is one of the few virtualization systems which supports SR-IOV.
But my experience is that it's hard to find a hardware combination which really lets you use it:
http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/40981
Signature · 3 lines
Regards, Troels Arvin <[email protected]> http://troels.arvin.dk/
Quoted 6 lines · Troels Arvin
Hello,
Quoted 1 line · Chris Friesen
Is anyone working on SR-IOV for KVM, specifically for networking?
KVM is one of the few virtualization systems which supports SR-IOV.
82576EB and a Dell R410 are working fine for me, but my host OS is Fedora 14, not RHEL6.
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686
David
Quoted 5 lines
But my experience is that it's hard to find a hardware combination which really lets you use it:
http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/40981
Quoted 2 lines · David Ahern
82576EB and a Dell R410 are working fine for me, but my host OS is Fedora 14, not RHEL6.
82576EB is a dual-port embedded NIC?
(The 82576 which is in "my" R710 is a quad-port PCIe card.)
Does your setup allow for SR-IOV without having to turn off ACS? (ACS is a security feature.)
Interesting thread, thanks.
Signature · 3 lines
Regards, Troels Arvin <[email protected]> http://troels.arvin.dk/
Quoted 5 lines · Troels Arvin
Quoted 2 lines · David Ahern
82576EB and a Dell R410 are working fine for me, but my host OS is Fedora 14, not RHEL6.
82576EB is a dual-port embedded NIC?
I have a quad-port PCIe card.
Quoted 5 lines
(The 82576 which is in "my" R710 is a quad-port PCIe card.)
Does your setup allow for SR-IOV without having to turn off ACS? (ACS is a security feature.)
I recall having to twiddle some BIOS settings - and doing so in iterations. I forget which settings exactly. I'll see if Dell has a tool to dump BIOS settings to a file that works with Fedora 14.
David
Quoted 6 lines
Interesting thread, thanks.
Quoted 4 lines · David Ahern
Quoted 2 lines
Does your setup allow for SR-IOV without having to turn off ACS? (ACS is a security feature.)
I recall having to twiddle some BIOS settings
I believe that if you turn off ACS, it's through a kernel module option.
Signature · 3 lines
Regards, Troels Arvin <[email protected]> http://troels.arvin.dk/