VFs go missing with latest kernel
Hi
I built the kernel.git tree (3.3-rc1) and noticed that even numbered VFs go missing for the last 2 ports of a quad nic card.
This is the relevant output from lspci for the stock Fedora 14 kernel (2.6.35.14-106):
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
For the 3.3.0-rc1 kernel I get:
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
Note that all of the even numbered VFs have disappeared. Accordingly, trying to launch the VM to which the VFs are assigned fails. git bisect pointed to this commit:
0224d663063d542b3d829706f3fcbd0f640f19b3 is the first bad commit commit 0224d663063d542b3d829706f3fcbd0f640f19b3
Author: Greg Rose <[email protected]> Date: Fri Oct 14 02:57:14 2011 +0000
igb: Check if subordinate VFs are assigned to virtual machines
Kvm and the Xen pci-back driver will set a flag in the virtual function pci device dev_flags when the VF is assigned to a guest VM. Before destroying subordinate VFs check to see if the flag is set and if so skip the call to pci_disable_sriov() to avoid system crashes.
Copy the maintainer for the Xen pci-back driver. Also CC'ing maintainers of all drivers found to call pci_disable_sriov().
Sure enough reverting the patch on 3.3.0-rc1 makes the VFs visible again.
I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the KVM_CAP_IOMMU check though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.
David
Quoted 91 lines
-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 2:54 PM
To: Rose, Gregory V
Cc: LKML
Subject: VFs go missing with latest kernelHi
I built the kernel.git tree (3.3-rc1) and noticed that even numbered VFs go missing for the last 2 ports of a quad nic card.
This is the relevant output from lspci for the stock Fedora 14 kernel (2.6.35.14-106):
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)For the 3.3.0-rc1 kernel I get:
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)Note that all of the even numbered VFs have disappeared. Accordingly, trying to launch the VM to which the VFs are assigned fails. git bisect pointed to this commit:
0224d663063d542b3d829706f3fcbd0f640f19b3is the first bad commit commit0224d663063d542b3d829706f3fcbd0f640f19b3Author: Greg Rose <[email protected]> Date: Fri Oct 14 02:57:14 2011 +0000igb: Check if subordinate VFs are assigned to virtual machines
Kvm and the Xen pci-back driver will set a flag in the virtual
function
pci devicedev_flagswhen the VF is assigned to a guest VM. Before destroying subordinate VFs check to see if the flag is set and if so skip the call topci_disable_sriov()to avoid system crashes.Copy the maintainer for the Xen pci-back driver. Also CC'ing maintainers of all drivers found to call
pci_disable_sriov().
Quoted 6 lines
Sure enough reverting the patch on 3.3.0-rc1 makes the VFs visible again.
I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the
KVM_CAP_IOMMUcheck though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.
I'll see if I can reproduce that. Is this all after a fresh boot with no unloading/reloading of the PF driver?
- Greg
Quoted 2 lines
David
Quoted 21 lines · Gregory V
Quoted 21 lines
Note that all of the even numbered VFs have disappeared. Accordingly, trying to launch the VM to which the VFs are assigned fails. git bisect pointed to this commit:
0224d663063d542b3d829706f3fcbd0f640f19b3is the first bad commit commit0224d663063d542b3d829706f3fcbd0f640f19b3Author: Greg Rose <[email protected]> Date: Fri Oct 14 02:57:14 2011 +0000igb: Check if subordinate VFs are assigned to virtual machines
Kvm and the Xen pci-back driver will set a flag in the virtual
function
pci devicedev_flagswhen the VF is assigned to a guest VM. Before destroying subordinate VFs check to see if the flag is set and if so skip the call topci_disable_sriov()to avoid system crashes.Copy the maintainer for the Xen pci-back driver. Also CC'ing maintainers of all drivers found to call
pci_disable_sriov().
Quoted 8 lines
Quoted 6 lines
Sure enough reverting the patch on 3.3.0-rc1 makes the VFs visible again.
I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the
KVM_CAP_IOMMUcheck though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.I'll see if I can reproduce that. Is this all after a fresh boot with no unloading/reloading of the PF driver?
Build kernel, reboot, run lspci - note missing VFs.
Quoted 8 lines
-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 3:05 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernel
Quoted 4 lines
Quoted 1 line
I'll see if I can reproduce that. Is this all after a fresh boot with
no unloading/reloading of the PF driver?
Build kernel, reboot, run lspci - note missing VFs.
Thanks. Would you mind sending me your kernel log output from dmesg?
- Greg
Quoted 4 lines · Gregory V
Quoted 4 lines
I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the
KVM_CAP_IOMMUcheck though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.
second problem was due to a CONFIG setting.
Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.
David
Quoted 11 lines
-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 4:31 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernelQuoted 1 line · Gregory V
Quoted 1 line
I am hitting some other problem trying to use the VF on 3.3.0-rc1 --
KVM
Quoted 1 line
Quoted 1 line
fails the
KVM_CAP_IOMMUcheck though the server has one (works fine
Quoted 8 lines
with
Quoted 2 lines
Quoted 2 lines
the older kernel) and it is built into the kernel. Debugging that problem now.
second problem was due to a CONFIG setting.
Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.
OK, thanks... I was staring at the dmesg output you sent me and it is odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.
Also, the dmesg output indicates 28 VFs are created but the lspci output you've sent only shows 14, even from the stock FC 14 kernel. So even in that case you seem to be missing 14 VFs from the lspci output.
So they're there at one point but then disappear. Very odd. I'm finishing up a few other things but will try to repro ASAP and get back to you with my results.
- Greg
------------
[ 81.432421] igb 0000:07:00.0: setting MAC 02:12:34:56:79:41 on VF 1 [ 81.432550] igb 0000:07:00.0: Reload the VF driver to make this change effective. [ 81.432667] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up. [ 81.432942] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device. [ 81.434947] igb 0000:07:00.0: setting MAC 02:12:34:56:79:42 on VF 2 [ 81.435060] igb 0000:07:00.0: Reload the VF driver to make this change effective. [ 81.435174] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up. [ 81.435450] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device. [ 81.437429] igb 0000:07:00.0: setting MAC 02:12:34:56:79:43 on VF 3 [ 81.437555] igb 0000:07:00.0: Reload the VF driver to make this change effective. [ 81.437670] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up. [ 81.437937] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device. [ 81.439904] igb 0000:07:00.0: setting MAC 02:12:34:56:79:44 on VF 4 [ 81.440016] igb 0000:07:00.0: Reload the VF driver to make this change effective. [ 81.440132] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up. [ 81.440400] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device. [ 81.442328] igb 0000:07:00.0: setting MAC 02:12:34:56:79:45 on VF 5 [ 81.442453] igb 0000:07:00.0: Reload the VF driver to make this change effective. [ 81.442567] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up. [ 81.442837] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device. [ 81.444921] igb 0000:07:00.0: setting MAC 02:12:34:56:79:46 on VF 6 [ 81.445035] igb 0000:07:00.0: Reload the VF driver to make this change effective. [ 81.445149] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up. [ 81.445417] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.
-------------------
Quoted 2 lines
David
Quoted 5 lines · Gregory V
Quoted 3 lines
Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.
OK, thanks... I was staring at the dmesg output you sent me and it is odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.
Ah, I think I confused you. The VFs for the first 2 ports (ie., the first 14 VFs) are seen fine; I only showed the VFs for the second 2 ports -- which are missing the even number.
So, in summary I am missing 7 VFs all associated with the even numbers on the last ports of a quad port card.
David
Quoted 16 lines
-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 4:50 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernelQuoted 5 lines · Gregory V
Quoted 3 lines
Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.
OK, thanks... I was staring at the dmesg output you sent me and it is
odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.
Quoted 6 lines
Ah, I think I confused you. The VFs for the first 2 ports (ie., the first 14 VFs) are seen fine; I only showed the VFs for the second 2 ports -- which are missing the even number.
So, in summary I am missing 7 VFs all associated with the even numbers on the last ports of a quad port card.
OK, thanks for the clarification. But it still shows that all 28 VFs exist while their MAC addresses are being set but then at some point subsequent to that 7 of them on the port belonging to PCI device 07:00.0 disappear for some reason.
I'll see what I can find.
Thanks,
- Greg
Quoted 2 lines
David
Quoted 31 lines
-----Original Message-----
From: Rose, Gregory V
Sent: Tuesday, January 31, 2012 5:03 PM
To: David Ahern
Cc: LKML
Subject: RE: VFs go missing with latest kernelQuoted 10 lines
-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 4:50 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernelQuoted 2 lines · Gregory V
Quoted 2 lines
Back to the missing VF problem: reverting the patch I mentioned
before
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
and my VM boots up fine with the even-numbered VFs assigned to it.
OK, thanks... I was staring at the dmesg output you sent me and it is
odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.
Quoted 6 lines
Ah, I think I confused you. The VFs for the first 2 ports (ie., the first 14 VFs) are seen fine; I only showed the VFs for the second 2 ports -- which are missing the even number.
So, in summary I am missing 7 VFs all associated with the even numbers on the last ports of a quad port card.
Quoted 6 lines
OK, thanks for the clarification. But it still shows that all 28 VFs exist while their MAC addresses are being set but then at some point subsequent to that 7 of them on the port belonging to PCI device 07:00.0 disappear for some reason.
I'll see what I can find.
I found this in the log file you sent me. I had missed it yesterday.
[ 15.835223] igb 0000:07:00.0: 7 pre-allocated VFs found - override max_vfs setting of 7 [ 15.835393] igb 0000:07:00.0: 7 VFs allocated
I think that must be a bug in the code that searches for VFs already allocated and is the source of your problem. I'll keep you updated on what I find but it has to be a bug in the VF device lookups.
- Greg
Quoted 6 lines · Gregory V
I found this in the log file you sent me. I had missed it yesterday.
[ 15.835223] igb 0000:07:00.0: 7 pre-allocated VFs found - override max_vfs setting of 7 [ 15.835393] igb 0000:07:00.0: 7 VFs allocatedI think that must be a bug in the code that searches for VFs already allocated and is the source of your problem. I'll keep you updated on what I find but it has to be a bug in the VF device lookups.
Confused. What is pre-allocating the VFs during boot? Looking at my rc-scripts I am only set MAC addresses at boot. No VMs have been started yet. This setup has worked fine with Fedora 14 and 3.0 kernels; it's really the move to 3.3-rc that I hit the problem.
David
Quoted 14 lines
-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Wednesday, February 01, 2012 9:58 AM
To: Rose, Gregory V
Cc: 'LKML'
Subject: Re: VFs go missing with latest kernelQuoted 3 lines · Gregory V
I found this in the log file you sent me. I had missed it yesterday.
[ 15.835223] igb 0000:07:00.0: 7 pre-allocated VFs found - override
max_vfssetting of 7Quoted 2 lines
[ 15.835393] igb 0000:07:00.0: 7 VFs allocated
Quoted 8 lines
Quoted 1 line
I think that must be a bug in the code that searches for VFs already
allocated and is the source of your problem. I'll keep you updated on what I find but it has to be a bug in the VF device lookups.
Confused. What is pre-allocating the VFs during boot? Looking at my rc-scripts I am only set MAC addresses at boot. No VMs have been started yet. This setup has worked fine with Fedora 14 and 3.0 kernels; it's really the move to 3.3-rc that I hit the problem.
Nothing is pre-allocating the VFs. That is the bug, the function I wrote to find pre-allocated VFs is messed up somehow. Since it (mistakenly) thinks it's already found pre-allocated VFs, it doesn't call pci_enable_sriov() to allocate them.
- Greg