Skip to contentSkip to conversation

VFs go missing with latest kernel

DISCUSSION David Ahern 2 participants 11 messages Jan 31, 2012 – Feb 1, 2012

lore
Share & tools
lore
#0 lore raw

Hi

I built the kernel.git tree (3.3-rc1) and noticed that even numbered VFs go missing for the last 2 ports of a quad nic card.

This is the relevant output from lspci for the stock Fedora 14 kernel (2.6.35.14-106):

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

For the 3.3.0-rc1 kernel I get:

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

Note that all of the even numbered VFs have disappeared. Accordingly, trying to launch the VM to which the VFs are assigned fails. git bisect pointed to this commit:

0224d663063d542b3d829706f3fcbd0f640f19b3 is the first bad commit commit 0224d663063d542b3d829706f3fcbd0f640f19b3

Author: Greg Rose <[email protected]>
Date:   Fri Oct 14 02:57:14 2011 +0000

igb: Check if subordinate VFs are assigned to virtual machines

Kvm and the Xen pci-back driver will set a flag in the virtual function pci device dev_flags when the VF is assigned to a guest VM. Before destroying subordinate VFs check to see if the flag is set and if so skip the call to pci_disable_sriov() to avoid system crashes.

Copy the maintainer for the Xen pci-back driver. Also CC'ing maintainers of all drivers found to call pci_disable_sriov().

Sure enough reverting the patch on 3.3.0-rc1 makes the VFs visible again.

I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the KVM_CAP_IOMMU check though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.

David

#1 lore raw
Quoted 91 lines

-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 2:54 PM
To: Rose, Gregory V
Cc: LKML
Subject: VFs go missing with latest kernel

Hi

I built the kernel.git tree (3.3-rc1) and noticed that even numbered VFs go missing for the last 2 ports of a quad nic card.

This is the relevant output from lspci for the stock Fedora 14 kernel (2.6.35.14-106):

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

For the 3.3.0-rc1 kernel I get:

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
08:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

Note that all of the even numbered VFs have disappeared. Accordingly, trying to launch the VM to which the VFs are assigned fails. git bisect pointed to this commit:

0224d663063d542b3d829706f3fcbd0f640f19b3 is the first bad commit commit 0224d663063d542b3d829706f3fcbd0f640f19b3

Author: Greg Rose <[email protected]>
Date:   Fri Oct 14 02:57:14 2011 +0000

igb: Check if subordinate VFs are assigned to virtual machines

Kvm and the Xen pci-back driver will set a flag in the virtual
function
pci device dev_flags when the VF is assigned to a guest VM. Before destroying subordinate VFs check to see if the flag is set and if so skip the call to pci_disable_sriov() to avoid system crashes.

Copy the maintainer for the Xen pci-back driver. Also CC'ing maintainers of all drivers found to call pci_disable_sriov().

Quoted 6 lines

Sure enough reverting the patch on 3.3.0-rc1 makes the VFs visible again.

I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the KVM_CAP_IOMMU check though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.

I'll see if I can reproduce that. Is this all after a fresh boot with no unloading/reloading of the PF driver?

- Greg

Quoted 2 lines

David

#2 lore raw
Quoted 21 lines · Gregory V
Quoted 21 lines

Note that all of the even numbered VFs have disappeared. Accordingly, trying to launch the VM to which the VFs are assigned fails. git bisect pointed to this commit:

0224d663063d542b3d829706f3fcbd0f640f19b3 is the first bad commit commit 0224d663063d542b3d829706f3fcbd0f640f19b3

Author: Greg Rose <[email protected]>
Date:   Fri Oct 14 02:57:14 2011 +0000

igb: Check if subordinate VFs are assigned to virtual machines

Kvm and the Xen pci-back driver will set a flag in the virtual
function
pci device dev_flags when the VF is assigned to a guest VM. Before destroying subordinate VFs check to see if the flag is set and if so skip the call to pci_disable_sriov() to avoid system crashes.

Copy the maintainer for the Xen pci-back driver. Also CC'ing maintainers of all drivers found to call pci_disable_sriov().

Quoted 8 lines
Quoted 6 lines

Sure enough reverting the patch on 3.3.0-rc1 makes the VFs visible again.

I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the KVM_CAP_IOMMU check though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.

I'll see if I can reproduce that. Is this all after a fresh boot with no unloading/reloading of the PF driver?

Build kernel, reboot, run lspci - note missing VFs.

#3 lore raw
Quoted 8 lines

-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 3:05 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernel

Quoted 4 lines
Quoted 1 line

I'll see if I can reproduce that. Is this all after a fresh boot with

no unloading/reloading of the PF driver?

Build kernel, reboot, run lspci - note missing VFs.

Thanks. Would you mind sending me your kernel log output from dmesg?

- Greg

#4 lore raw
In reply to Rose, Gregory V
Quoted 4 lines · Gregory V
Quoted 4 lines

I am hitting some other problem trying to use the VF on 3.3.0-rc1 -- KVM fails the KVM_CAP_IOMMU check though the server has one (works fine with the older kernel) and it is built into the kernel. Debugging that problem now.

second problem was due to a CONFIG setting.

Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.

David

#5 lore raw
Quoted 11 lines

-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 4:31 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernel

Quoted 1 line · Gregory V
Quoted 1 line

I am hitting some other problem trying to use the VF on 3.3.0-rc1 --

KVM

Quoted 1 line
Quoted 1 line

fails the KVM_CAP_IOMMU check though the server has one (works fine

Quoted 8 lines

with

Quoted 2 lines
Quoted 2 lines

the older kernel) and it is built into the kernel. Debugging that problem now.

second problem was due to a CONFIG setting.

Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.

OK, thanks... I was staring at the dmesg output you sent me and it is odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.

Also, the dmesg output indicates 28 VFs are created but the lspci output you've sent only shows 14, even from the stock FC 14 kernel. So even in that case you seem to be missing 14 VFs from the lspci output.

So they're there at one point but then disappear. Very odd. I'm finishing up a few other things but will try to repro ASAP and get back to you with my results.

- Greg

------------

[   81.432421] igb 0000:07:00.0: setting MAC 02:12:34:56:79:41 on VF 1
[   81.432550] igb 0000:07:00.0: Reload the VF driver to make this change effective.
[   81.432667] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up.
[   81.432942] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.
[   81.434947] igb 0000:07:00.0: setting MAC 02:12:34:56:79:42 on VF 2
[   81.435060] igb 0000:07:00.0: Reload the VF driver to make this change effective.
[   81.435174] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up.
[   81.435450] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.
[   81.437429] igb 0000:07:00.0: setting MAC 02:12:34:56:79:43 on VF 3
[   81.437555] igb 0000:07:00.0: Reload the VF driver to make this change effective.
[   81.437670] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up.
[   81.437937] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.
[   81.439904] igb 0000:07:00.0: setting MAC 02:12:34:56:79:44 on VF 4
[   81.440016] igb 0000:07:00.0: Reload the VF driver to make this change effective.
[   81.440132] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up.
[   81.440400] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.
[   81.442328] igb 0000:07:00.0: setting MAC 02:12:34:56:79:45 on VF 5
[   81.442453] igb 0000:07:00.0: Reload the VF driver to make this change effective.
[   81.442567] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up.
[   81.442837] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.
[   81.444921] igb 0000:07:00.0: setting MAC 02:12:34:56:79:46 on VF 6
[   81.445035] igb 0000:07:00.0: Reload the VF driver to make this change effective.
[   81.445149] igb 0000:07:00.0: The VF MAC address has been set, but the PF device is not up.
[   81.445417] igb 0000:07:00.0: Bring the PF device up before attempting to use the VF device.

-------------------

Quoted 2 lines

David

#6 lore raw
Quoted 5 lines · Gregory V
Quoted 3 lines

Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.

OK, thanks... I was staring at the dmesg output you sent me and it is odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.

Ah, I think I confused you. The VFs for the first 2 ports (ie., the first 14 VFs) are seen fine; I only showed the VFs for the second 2 ports -- which are missing the even number.

So, in summary I am missing 7 VFs all associated with the even numbers on the last ports of a quad port card.

David

#7 lore raw
Quoted 16 lines

-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 4:50 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernel

Quoted 5 lines · Gregory V
Quoted 3 lines

Back to the missing VF problem: reverting the patch I mentioned before and my VM boots up fine with the even-numbered VFs assigned to it.

OK, thanks... I was staring at the dmesg output you sent me and it is

odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.

Quoted 6 lines

Ah, I think I confused you. The VFs for the first 2 ports (ie., the first 14 VFs) are seen fine; I only showed the VFs for the second 2 ports -- which are missing the even number.

So, in summary I am missing 7 VFs all associated with the even numbers on the last ports of a quad port card.

OK, thanks for the clarification. But it still shows that all 28 VFs exist while their MAC addresses are being set but then at some point subsequent to that 7 of them on the port belonging to PCI device 07:00.0 disappear for some reason.

I'll see what I can find.

Thanks,

- Greg

Quoted 2 lines

David

#8 lore raw
Quoted 31 lines

-----Original Message-----
From: Rose, Gregory V
Sent: Tuesday, January 31, 2012 5:03 PM
To: David Ahern
Cc: LKML
Subject: RE: VFs go missing with latest kernel

Quoted 10 lines

-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Tuesday, January 31, 2012 4:50 PM
To: Rose, Gregory V
Cc: LKML
Subject: Re: VFs go missing with latest kernel

Quoted 2 lines · Gregory V
Quoted 2 lines

Back to the missing VF problem: reverting the patch I mentioned

before

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

and my VM boots up fine with the even-numbered VFs assigned to it.

OK, thanks... I was staring at the dmesg output you sent me and it is

odd because the even numbered VFs that go missing are there when you set their MAC addresses. See output below the dotted line after my reply.

Quoted 6 lines

Ah, I think I confused you. The VFs for the first 2 ports (ie., the first 14 VFs) are seen fine; I only showed the VFs for the second 2 ports -- which are missing the even number.

So, in summary I am missing 7 VFs all associated with the even numbers on the last ports of a quad port card.

Quoted 6 lines

OK, thanks for the clarification. But it still shows that all 28 VFs exist while their MAC addresses are being set but then at some point subsequent to that 7 of them on the port belonging to PCI device 07:00.0 disappear for some reason.

I'll see what I can find.

I found this in the log file you sent me. I had missed it yesterday.

[   15.835223] igb 0000:07:00.0: 7 pre-allocated VFs found - override max_vfs setting of 7
[   15.835393] igb 0000:07:00.0: 7 VFs allocated

I think that must be a bug in the code that searches for VFs already allocated and is the source of your problem. I'll keep you updated on what I find but it has to be a bug in the VF device lookups.

- Greg

#9 lore raw
Quoted 6 lines · Gregory V

I found this in the log file you sent me. I had missed it yesterday.

[   15.835223] igb 0000:07:00.0: 7 pre-allocated VFs found - override max_vfs setting of 7
[   15.835393] igb 0000:07:00.0: 7 VFs allocated

I think that must be a bug in the code that searches for VFs already allocated and is the source of your problem. I'll keep you updated on what I find but it has to be a bug in the VF device lookups.

Confused. What is pre-allocating the VFs during boot? Looking at my rc-scripts I am only set MAC addresses at boot. No VMs have been started yet. This setup has worked fine with Fedora 14 and 3.0 kernels; it's really the move to 3.3-rc that I hit the problem.

David

#10 lore raw
Quoted 14 lines

-----Original Message-----
From: David Ahern [mailto:[email protected]]
Sent: Wednesday, February 01, 2012 9:58 AM
To: Rose, Gregory V
Cc: 'LKML'
Subject: Re: VFs go missing with latest kernel

Quoted 3 lines · Gregory V

I found this in the log file you sent me. I had missed it yesterday.

[   15.835223] igb 0000:07:00.0: 7 pre-allocated VFs found - override

max_vfs setting of 7

Quoted 2 lines
[   15.835393] igb 0000:07:00.0: 7 VFs allocated
Quoted 8 lines
Quoted 1 line

I think that must be a bug in the code that searches for VFs already

allocated and is the source of your problem. I'll keep you updated on what I find but it has to be a bug in the VF device lookups.

Confused. What is pre-allocating the VFs during boot? Looking at my rc-scripts I am only set MAC addresses at boot. No VMs have been started yet. This setup has worked fine with Fedora 14 and 3.0 kernels; it's really the move to 3.3-rc that I hit the problem.

Nothing is pre-allocating the VFs. That is the bug, the function I wrote to find pre-allocated VFs is messed up somehow. Since it (mistakenly) thinks it's already found pre-allocated VFs, it doesn't call pci_enable_sriov() to allocate them.

- Greg