Skip to contentSkip to conversation

PCI: Init NumVFs register to zero in sriov_init()

PATCH Applied ethan.zhao 3 participants 9 messages Nov 6, 2013 – Nov 22, 2013 PCI

v1 of 2 Applied Latest is v2 →

lore
Share & tools
lore

ethan.zhao

author
#0 lore raw

PATCH 1/1 PCI: Init NumVFs register to zero in sriov_init()

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
~

---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -0
index de8ffac..a4941ad 100644
@@ -439,6 +439,8 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
 found:
 	pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl);
+	/* VF Enable is cleared, so we could init the NumVFs register to 0 */
+	pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, 0);
 	pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
 	pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
 	if (!offset || (total > 1 && !stride))
Signature · 1 line
1.8.3.4 (Apple Git-47)
author
#1 lore raw

Yinghai,
Could you help to take a look, we need it in our downstream kernel.

Thanks,
Ethan

Quoted 35 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
~

---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -1
index de8ffac..a4941ad 100644
@@ -439,6 +439,8 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
 found:
        pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl);
+       /* VF Enable is cleared, so we could init the NumVFs register to 0 */
+       pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, 0);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
        if (!offset || (total > 1 && !stride))
--

1.8.3.4 (Apple Git-47)

#2 lore raw
Quoted 4 lines · Ethan Zhao
Quoted 3 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

Does BIOS need to clear it?

Quoted 11 lines
Quoted 11 lines

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
~

just display problem?

Quoted 11 lines
Quoted 11 lines
---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +0 -0
index de8ffac..a4941ad 100644
@@ -439,6 +439,8 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
Quoted 7 lines

drivers/pci/iov.c static int sriov_init(struct pci_dev *dev, int pos) · line 446

Quoted 7 lines
drivers/pci/iov.c +2 -0
 found:
        pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl);
+       /* VF Enable is cleared, so we could init the NumVFs register to 0 */
+       pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, 0);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
        if (!offset || (total > 1 && !stride))

Thanks

Yinghai

author
#3 lore raw
Quoted 7 lines · Yinghai Lu
Quoted 4 lines · Ethan Zhao
Quoted 3 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

Does BIOS need to clear it?

No, don't think BIOS need to clear it, according to the SR-IOV and PCI specification, BIOS hasn't the responsibility to clear NumVFs register
or not. "The initial value of NumVFs is undefined." (SINGLE ROOT I/O
VIRTUALIZATION AND SHARING SPECIFICATION, REV. 1.1 chapter 3.3.7 ). In fact, Linux doesn't trust the PCI_SRIOV_XX registers initial value left by BIOS, exactly, it did in iov.c , Linux will re-initialize the PCI_SRIOV_XX registers' value whatever their original value left by BIOS.

static int sriov_init(struct pci_dev *dev, int pos) {
... ...
pci_read_config_word(dev, pos + PCI_SRIOV_CTRL, &ctrl); if (ctrl & PCI_SRIOV_CTRL_VFE) { pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, 0); ssleep(1);
}

... ...
ctrl = 0;
... ...
found:
pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl); ... ...
}

If so, why leave value of NumVFs register un-initialized, till be exposed to user via lspci etc.

Quoted 14 lines
Quoted 13 lines
Quoted 13 lines

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
~

Quoted 1 line

just display problem?

So far, only catch the issue via lspci, "seems' only confuses user, the VFs are enabled to 64. in fact, they are disabled. but it is not bug of lspci, binary dump of lspci could prove it.

Thanks,
Ethan

Quoted 23 lines
Quoted 12 lines
Quoted 12 lines
---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +0 -0
index de8ffac..a4941ad 100644
@@ -439,6 +439,8 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
Quoted 7 lines

drivers/pci/iov.c static int sriov_init(struct pci_dev *dev, int pos) · line 446

Quoted 7 lines
drivers/pci/iov.c +2 -0
 found:
        pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl);
+       /* VF Enable is cleared, so we could init the NumVFs register to 0 */
+       pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, 0);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
        if (!offset || (total > 1 && !stride))

Thanks

Yinghai

maintainer
#4 lore raw
In reply to ethan.zhao

[+cc linux-pci]

Quoted 24 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
~

---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +0 -0
index de8ffac..a4941ad 100644

drivers/pci/iov.c static int sriov_init(struct pci_dev *dev, int pos) · line 446

Quoted 8 lines
drivers/pci/iov.c +2 -0
@@ -439,6 +439,8 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
 found:
        pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl);
+       /* VF Enable is cleared, so we could init the NumVFs register to 0 */
+       pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, 0);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
        if (!offset || (total > 1 && !stride))

I'm not 100% sure we should always disable SR-IOV when we enumerate a PF, but as long as we do, it seems OK to me to clear NumVFs at the same time.

We already clear NumVFs at other places where we disable SR-IOV (sriov_enable() failure path and sriov_disable()).

Bjorn

maintainer
#5 lore raw
Quoted 16 lines · Bjorn Helgaas

[+cc linux-pci]

Quoted 13 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !

Did you mean to show lspci output from before SR-IOV was enabled? It looks like SR-IOV is enabled here, so I don't think your patch would change this output at all.

Bjorn

author
#6 lore raw

Bjorn,
To be honest, the output above is picked up from other guy's mail,
because I composed the patch mail at home and left the real bug material at company, they 'really' hit an issue confused by lspci output while SR-IOV is initialized to disabled .
There is an error in above output as description that not made on
purpose, sorry again, the
IOVCtl: Enable+
should be
IOVCtl: Enable-

Current iov.c code, in fact, initializing the SR-IOV to disabled whatever BIOS setup the PCI_SRIOV_XX registers.

Thanks,
Ethan

Quoted 23 lines · Bjorn Helgaas
Quoted 16 lines · Bjorn Helgaas

[+cc linux-pci]

Quoted 13 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !

Did you mean to show lspci output from before SR-IOV was enabled? It looks like SR-IOV is enabled here, so I don't think your patch would change this output at all.

Bjorn

author
#7 lore raw
In reply to Bjorn Helgaas

Bjorn,
I revised the description part with the original bug material as
below, help to take a look, will send V2 back to home, SMTP blocked by company network.

PCI: Init NumVFs register to zero in sriov_init()

Though no specification about NumVFs register initial value after
POST, to void the confusion
lspci output as following before VF was enabled, we should clear
the NumVFs value left by BIOS

    to zero:

    #lspci -vvvxxx -s 13:00.0

13:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+
Network Connection (rev 01) Subsystem: Oracle/SUN Ethernet Server
Adapter X520-2
~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
140: 03 00 01 15 c4 24 c8 ff ff 21 1b 00 00 00 00 00
150: 0e 00 01 16 00 01 00 00 00 00 00 00 00 00 00 00
160: 10 00 01 00 00 00 00 00 10 00 00 00 40 00 40 00

^ ^Total VFs

Initial VFs
170: 40 00 00 00 80 00 02 00 00 00 ed 10 53 05 00 00
^Number of VFs

Thanks,
Ethan

Quoted 23 lines · Bjorn Helgaas
Quoted 16 lines · Bjorn Helgaas

[+cc linux-pci]

Quoted 13 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !

Did you mean to show lspci output from before SR-IOV was enabled? It looks like SR-IOV is enabled here, so I don't think your patch would change this output at all.

Bjorn

maintainer Applied
#8 lore raw
In reply to ethan.zhao
Quoted 14 lines · ethan.zhao

Though no specification about NumVFs register initial value after POST, to void the confusion lspci output as following before VF was enabled, we should clear the NumVFs value left by BIOS to zero:

$lspci -vvv -s 03:00.0
Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) ~
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
^dazed !
~

Quoted 1 line

Applied to my pci/misc branch for v3.14, thanks!

Bjorn

Quoted 20 lines
---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -1
index de8ffac..a4941ad 100644
@@ -439,6 +439,8 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
 found:
        pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, ctrl);
+       /* VF Enable is cleared, so we could init the NumVFs register to 0 */
+       pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, 0);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
        pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
        if (!offset || (total > 1 && !stride))
--

1.8.3.4 (Apple Git-47)