New: igb VF can't work in KVM guest
https://bugzilla.kernel.org/show_bug.cgi?id=55421
Summary: igb VF can't work in KVM guest
Product: Virtualization
Version: unspecified
Kernel Version: 3.9.0-rc1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No
Environment:
------------
Host OS (ia32/ia32e/IA64):ia32e
Guest OS (ia32/ia32e/IA64):ia32e
Guest OS Type (Linux/Windows):Linux
kvm.git next branch Commit:5da596078f915a62e39a20e582308eab91b88c9a qemu-kvm uq/master branch Commit:3e41a753551a906dd9ed66fb0fc34167a6af3ba0 Host Kernel Version:3.9.0-rc1
Hardware:SNB-EP, WSM-EP
Bug detailed description:
--------------------------
After assigning a igb VF to a guest or hot-plugging a VF to a guest, the VF cannot work in the guest.
Note:
1. The VF of Intel 82599 NIC (ixgbe) can work in guest
2. The PF of Intel 82576 NIC or Intel I350 NIC (using igb driver) can work in guest
This should be a kvm bug.
kvm + qemu-kvm = result 5da59607 + 3e41a753 = bad 3ab66e8a + 3e41a753 = good
Reproduce steps:
----------------
1.start up a KVM host
2.assign a VF to a guest in qemu commandline:
qemu-system-x86_64 --enable-kvm -m 1024 -smp 2 -device pci-assign,host=05:10.0 -net none -hda rhel6u3.img
Current result:
----------------
the VF of cannot work in guest
Expected result:
----------------
the VF can work in the guest
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 55421] igb VF can't work in KVM guest
https://bugzilla.kernel.org/show_bug.cgi?id=55421
--- Comment #1 from Jay Ren <[email protected]> 2013-03-18 15:11:16 --- Created an attachment (id=95741)
--> (https://bugzilla.kernel.org/attachment.cgi?id=95741)
host dmesg
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 55421] igb VF can't work in KVM guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=55421
--- Comment #2 from Jay Ren <[email protected]> 2013-03-18 15:12:02 --- Created an attachment (id=95751)
--> (https://bugzilla.kernel.org/attachment.cgi?id=95751)
guest dmesg
in the dmesg in guest:
igbvf 0000:00:03.0: irq 26 for MSI/MSI-X
igbvf 0000:00:03.0: Invalid MAC Address: 00:00:00:00:00:00
igbvf: probe of 0000:00:03.0 failed with error -5
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 55421] igb VF can't work in KVM guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=55421
--- Comment #3 from Jay Ren <[email protected]> 2013-03-18 15:12:33 --- Created an attachment (id=95761)
--> (https://bugzilla.kernel.org/attachment.cgi?id=95761)
lspci info of the igbvf in guest
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 55421] igb VF can't work in KVM guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=55421
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected], | |[email protected]
--- Comment #4 from Alex Williamson <[email protected]> 2013-03-21 21:29:09 --- Further bisected to:
commit 5ac6f91d39e0884813dc010e14552143cd1d0d8b
Author: Mitch A Williams <[email protected]> Date: Fri Jan 18 08:57:20 2013 +0000
igb: Don't give VFs random MAC addresses
If the user has not assigned a MAC address to a VM, then don't give it a random one. Instead, just give it zeros and let it figure out what to do with them.
So, for whatever reason we no longer assign a random MAC address when using the device in a VM (but we do still use one if attached to igbvf in the host). I expect we'll eventually see this on all the Intel SR-IOV NICs. The solution is to use the ip command to assign the VF a valid MAC address prior to using it with KVM. I'll let those who made the change defend it further if they wish.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
Re: [Bug 55421] igb VF can't work in KVM guest
Mitch,
Bugzilla wouldn't let me add you to the CC, so FYI... Thanks,
Alex
Quoted 39 lines · On Thu, 2013-03-21 at 21:29 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=55421
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------CC| |[email protected], | |[email protected]--- Comment #4 from Alex Williamson <[email protected]> 2013-03-21 21:29:09 --- Further bisected to:
commit
5ac6f91d39e0884813dc010e14552143cd1d0d8bAuthor: Mitch A Williams <[email protected]> Date: Fri Jan 18 08:57:20 2013 +0000igb: Don't give VFs random MAC addresses
If the user has not assigned a MAC address to a VM, then don't give it a random one. Instead, just give it zeros and let it figure out what to do with them.
Signed-off-by Mitch Williams <[email protected]>CC Andy Gospodarek <[email protected]>CC Stefan Assmann <[email protected]>Tested-by Aaron Brown <[email protected]>Tested-by Stefan Assmann <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>So, for whatever reason we no longer assign a random MAC address when using the device in a VM (but we do still use one if attached to igbvf in the host). I expect we'll eventually see this on all the Intel SR-IOV NICs. The solution is to use the ip command to assign the VF a valid MAC address prior to using it with KVM. I'll let those who made the change defend it further if they wish.
RE: [Bug 55421] igb VF can't work in KVM guest
Thanks for the FYI, Alex. When we pushed this patch upstream, there was a corresponding patch to igbvf that was pushed at the same time. With this patch, the VFs detect the zero address and generate a random MAC address themselves. The reason for this change is to make the VFs play nicely with udev. The issue was originally raised by Stefan Assmann.
With older drivers in the guest, this will cause the issue you noted. Distros should cherry-pick the corresponding VF patch.
-Mitch
Quoted 68 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, March 21, 2013 2:32 PM
To: Williams, Mitch A
Cc: [email protected]
Subject: Re: [Bug 55421] igb VF can't work in KVM guestMitch,
Bugzilla wouldn't let me add you to the CC, so FYI... Thanks,
Alex
Quoted 7 lines · On Thu, 2013-03-21 at 21:29 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=55421
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 1 line
CC|Quoted 1 line
|Quoted 5 lines
--- Comment #4 from Alex Williamson <[email protected]> 2013-03-
21 21:29:09 ---
Quoted 9 lines
Further bisected to:
commit
5ac6f91d39e0884813dc010e14552143cd1d0d8bAuthor: Mitch A Williams <[email protected]> Date: Fri Jan 18 08:57:20 2013 +0000igb: Don't give VFs random MAC addresses
If the user has not assigned a MAC address to a VM, then don't give it
a
Quoted 1 line
random one. Instead, just give it zeros and let it figure out what to
do
Quoted 10 lines
with them.
Signed-off-by Mitch Williams <[email protected]>CC Andy Gospodarek <[email protected]>CC Stefan Assmann <[email protected]>Tested-by Aaron Brown <[email protected]>Tested-by Stefan Assmann <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>So, for whatever reason we no longer assign a random MAC address when
using the
Quoted 1 line
device in a VM (but we do still use one if attached to igbvf in the host).
I
Quoted 1 line
expect we'll eventually see this on all the Intel SR-IOV NICs. The
solution is
Quoted 1 line
to use the ip command to assign the VF a valid MAC address prior to using
it
Quoted 1 line
with KVM. I'll let those who made the change defend it further if they
wish.
Quoted 1 line
Re: [Bug 55421] igb VF can't work in KVM guest
Quoted 9 lines · Mitch A
Thanks for the FYI, Alex. When we pushed this patch upstream, there was a corresponding patch to igbvf that was pushed at the same time. With this patch, the VFs detect the zero address and generate a random MAC address themselves. The reason for this change is to make the VFs play nicely with udev. The issue was originally raised by Stefan Assmann.
With older drivers in the guest, this will cause the issue you noted. Distros should cherry-pick the corresponding VF patch.
I agree the random MAC doesn't play nicely with udev, although I have found that my guest stops adding new persistent net rules after eth680 ;) Arguably anyone using VFs for more than testing should already be assigning a stable MAC address, but I don't really think a guest driver update across every possible guest OS is all that practical. At least that explains why igbvf in the host is unaffected though. Should we expect a Code 10 (device cannot start) from the Windows igbvf driver when it finds a zero'd MAC? Thanks,
Alex
Quoted 67 lines
Quoted 67 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, March 21, 2013 2:32 PM
To: Williams, Mitch A
Cc: [email protected]
Subject: Re: [Bug 55421] igb VF can't work in KVM guestMitch,
Bugzilla wouldn't let me add you to the CC, so FYI... Thanks,
Alex
Quoted 7 lines · On Thu, 2013-03-21 at 21:29 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=55421
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 1 line
CC|Quoted 1 line
|Quoted 5 lines
--- Comment #4 from Alex Williamson <[email protected]> 2013-03-
21 21:29:09 ---
Quoted 9 lines
Further bisected to:
commit
5ac6f91d39e0884813dc010e14552143cd1d0d8bAuthor: Mitch A Williams <[email protected]> Date: Fri Jan 18 08:57:20 2013 +0000igb: Don't give VFs random MAC addresses
If the user has not assigned a MAC address to a VM, then don't give it
a
Quoted 1 line
random one. Instead, just give it zeros and let it figure out what to
do
Quoted 10 lines
with them.
Signed-off-by Mitch Williams <[email protected]>CC Andy Gospodarek <[email protected]>CC Stefan Assmann <[email protected]>Tested-by Aaron Brown <[email protected]>Tested-by Stefan Assmann <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>So, for whatever reason we no longer assign a random MAC address when
using the
Quoted 1 line
device in a VM (but we do still use one if attached to igbvf in the host).
I
Quoted 1 line
expect we'll eventually see this on all the Intel SR-IOV NICs. The
solution is
Quoted 1 line
to use the ip command to assign the VF a valid MAC address prior to using
it
Quoted 1 line
with KVM. I'll let those who made the change defend it further if they
wish.
Quoted 1 line
[Bug 55421] igb VF can't work in KVM guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=55421
Jay Ren <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |INVALID--- Comment #5 from Jay Ren <[email protected]> 2013-04-22 14:14:26 --- As this is the latest igb drive change, I'll close this bug.
There are two methods to work around or fix this issue.
1. update igbvf driver or kernel of the guest
2. use ip command to set VF's MAC before assigning to a guest (e.g. ip link set eth0 vf 0 mac 00:1E:67:65:93:01 )
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 55421] igb VF can't work in KVM guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=55421
Jay Ren <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |CLOSED
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.