New driver vxge for Neterion's X3100 series 10 GbE PCIe adapter
Subsystems (2)
This is a release of a new network driver, "vxge", for our latest PCIe based hardware - The X3100 10GbE Server/Storage Adapter. The X3100 ASIC supports four modes of operation, configurable via firmware -
Single function mode
Multi function mode
SRIOV mode
MRIOV mode
The driver patches series will follow this email. This driver has undergone significant testing for the past six months in all four modes of operation, and is very stable. We would appreciate the community review and comments on this driver.
The modes, besides single function mode, are oriented towards Server I/O virtualization and/or I/O sharing (see PCI SIG SR IOV and MR IOV specs for reference), although they can be used on a single non-virtualized server as well for instance, to run workloads that would typically benefit from using separate network cards. In these scenarios, X3100 can replace large number of GbE NICs without any system or network changes (outside of L2 driver/hardware), while each physical NIC will be able run up to 10 GbE instead of 1 GbE.
Major features include -
Virtual ethernet bridge
Multiqueue enabled
Service level guarantees per queue
Dual port with integrated IEEE 802.3ad link aggregation Tcp/Udp/IP stateless offloads -
TCP/UDP/IPv4/IPv6 checksum offload, TSO.
Large receive offload
Receive traffic hashing
MSI-X interrupts
Multiple tx and rx queues with number of steering options
A note on the different modes of operation -
Single-function mode: From Linux stack perspective, the adapter is a typical multi-queue 10GbE pci-e netdev interface (driven by the submitted vxge driver).
Multi-function mode: From Linux stack perspective, the adapter is a multi-function pci-e device where each function is a multi-queue pci-e netdev interface. This mode has some applications in native Linux environments, but it is primarily designed for use in hypervisors that do not yet support SR IOV pci-e extensions. In fact, the functionality in this mode is virtually identical to the SR IOV mode described below, except SR IOV support in Linux is not required/expected.
Quoted 1 line
From vxge driver perspective, each x3100 function is an independent NIC.
The functions share 10GbE link and pci-e bus, but hardly anything else inside the ASIC. Features like independent hw reset, statistics, bandwidth/priority allocation and guarantees, etc. are supported. Also, typical L2 functions like LSO, interrupt moderation, etc are supported independently on each function.
In other words, direct hardware access with x3100 doesnctions for Direct Hardware
Access to different Guests or Servers - while keeping traditional privileged operations (VLAN membership, etc) under privileged domain control.
SR IOV mode: Once BIOS and/or OS support SR IOV pci-e extensions, from Linux stack perspective the adapter will look like the multi-function mode in #2 above. vxge driver will still think that each x3100 function is an independent x3100 pci-e device.
MR IOV mode: This mode is designed to support sharing single pci-e 10GbE device across multiple independent servers (see PCI SIG MR IOV spec and/or presentations for reference). In PCI SIG terms, in this mode the X3100 works as a shared IOV 10GbE endpoint that assumes the presence of a pci-e switch. To the Linux stack on each server, the X3100 function (assigned to the server by the pci-e switch) will look like a separate X3100 adapter driven by the vxge driver.
The patch set consists of the the following files -
Documentation/networking/vxge.txt | 476 +++ MAINTAINERS | 7 +- drivers/net/Kconfig | 18 + drivers/net/Makefile | 1 + drivers/net/vxge/Makefile | 9 + drivers/net/vxge/vxge-config.c | 7576 +++++++++++++++++++++++++++++++++++++ drivers/net/vxge/vxge-config.h | 3323 ++++++++++++++++ drivers/net/vxge/vxge-ethtool.c | 1450 +++++++ drivers/net/vxge/vxge-ethtool.h | 73 + drivers/net/vxge/vxge-main.c | 5204 +++++++++++++++++++++++++ drivers/net/vxge/vxge-main.h | 623 +++ drivers/net/vxge/vxge-reg.h | 5659 +++++++++++++++++++++++++++ drivers/net/vxge/vxge-traffic.c | 2969 +++++++++++++++ drivers/net/vxge/vxge-traffic.h | 2808 ++++++++++++++ drivers/net/vxge/vxge-version.h | 24 + include/linux/pci_ids.h | 2 + 16 files changed, 30220 insertions(+), 2 deletions(-)
Regards,
Ram Vepa
David Miller
maintainerPlease resubmit this with the feedback you've received addressed.
Also, no need to CC: jgarzik, he doesn't handle network driver patch submissions any more.
Thanks for the feedback. We'll make the changes, retest and then resubmit.
Ram
Quoted 2 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
Quoted 5 lines
On Behalf Of David Miller
Sent: Saturday, March 14, 2009 1:58 PM
To: [email protected]
Cc: [email protected]
Subject: Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10
GbE
Quoted 11 lines
PCIe adapter
Please resubmit this with the feedback you've received addressed.
Also, no need to CC: jgarzik, he doesn't handle network driver patch submissions any more.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Ram,
Quoted 2 lines · Ramkrishna Vepa
Thanks for the feedback. We'll make the changes, retest and then resubmit.
BTW I got the following mail delivery error on my emails to you:
From: [email protected]
Date: Sat, 14 Mar 2009 15:23:49 -0400
Subject: Delivery Status Notification (Failure)
Unable to deliver message to the following recipients, because the message was f orwarded more than the maximum allowed times. This could indicate a mail loop.
-Bill
Hmmm... Not sure why you got this error message. Will check with our IT guys.
Thanks,
Ram
Quoted 6 lines
-----Original Message-----
From: Bill Fink [mailto:[email protected]]
Sent: Sunday, March 15, 2009 4:30 PM
To: Ramkrishna Vepa
Cc: David Miller; [email protected]
Subject: Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10
GbE
Quoted 9 lines
PCIe adapter
Hi Ram,
Quoted 2 lines · Ramkrishna Vepa
Thanks for the feedback. We'll make the changes, retest and then resubmit.
Quoted 7 lines
BTW I got the following mail delivery error on my emails to you:
From: [email protected]
Date: Sat, 14 Mar 2009 15:23:49 -0400
Subject: Delivery Status Notification (Failure)Unable to deliver message to the following recipients, because the
message
Quoted 2 lines
was f
orwarded more than the maximum allowed times. This could indicate a
Quoted 5 lines
loop.
-Bill
Quoted 28 lines · Ramkrishna Vepa
This is a release of a new network driver, "vxge", for our latest PCIe based hardware - The X3100 10GbE Server/Storage Adapter. The X3100 ASIC supports four modes of operation, configurable via firmware -
Single function mode
Multi function mode
SRIOV mode
MRIOV modeThe driver patches series will follow this email. This driver has undergone significant testing for the past six months in all four modes of operation, and is very stable. We would appreciate the community review and comments on this driver.
The modes, besides single function mode, are oriented towards Server I/O virtualization and/or I/O sharing (see PCI SIG SR IOV and MR IOV specs for reference), although they can be used on a single non-virtualized server as well for instance, to run workloads that would typically benefit from using separate network cards. In these scenarios, X3100 can replace large number of GbE NICs without any system or network changes (outside of L2 driver/hardware), while each physical NIC will be able run up to 10 GbE instead of 1 GbE.
A note on the different modes of operation -
Single-function mode: From Linux stack perspective, the adapter is a typical multi-queue 10GbE pci-e netdev interface (driven by the submitted vxge driver).
Quoted 7 lines
Multi-function mode: From Linux stack perspective, the adapter is a multi-function pci-e device where each function is a multi-queue pci-e netdev interface. This mode has some applications in native Linux environments, but it is primarily designed for use in hypervisors that do not yet support SR IOV pci-e extensions. In fact, the functionality in this mode is virtually identical to the SR IOV mode described below, except SR IOV support in Linux is not required/expected.
Xen upstream already supports the SR-IOV, and the native Linux and KVM will be supporting it too when 2.6.30 comes out.
Intel 82576 driver has been patched to enable the SR-IOV capability:
http://patchwork.kernel.org/patch/8063/
http://patchwork.kernel.org/patch/8064/
http://patchwork.kernel.org/patch/8065/
http://patchwork.kernel.org/patch/8066/
Thought one of these patches uses the sysfs interface to receive the NumVFs from user space, which is deprecated, rest of them still clearly demonstrate how to convert the traditional PCI NIC driver to the `Physical Function' driver by using the SR-IOV API.
Quoted 18 lines
Quoted 2 lines
From vxge driver perspective, each x3100 function is an independent NIC.
The functions share 10GbE link and pci-e bus, but hardly anything else inside the ASIC. Features like independent hw reset, statistics, bandwidth/priority allocation and guarantees, etc. are supported. Also, typical L2 functions like LSO, interrupt moderation, etc are supported independently on each function.
In other words, direct hardware access with x3100 doesnctions for Direct Hardware
Access to different Guests or Servers - while keeping traditional privileged operations (VLAN membership, etc) under privileged domain control.SR IOV mode: Once BIOS and/or OS support SR IOV pci-e extensions, from Linux stack perspective the adapter will look like the multi-function mode in #2 above. vxge driver will still think that each x3100 function is an independent x3100 pci-e device.
RE: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10 GbEPCIe adapter
Quoted 4 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
Quoted 9 lines
On Behalf Of Yu Zhao
Sent: Monday, March 30, 2009 11:14 PM
To: Ramkrishna Vepa
Cc: Netdev; David Miller
Subject: Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10 GbEPCIe adapterQuoted 1 line · Ramkrishna Vepa
This is a release of a new network driver, "vxge", for our latest
PCIe
Quoted 9 lines
based
Quoted 1 line
hardware - The X3100 10GbE Server/Storage Adapter. The X3100 ASIC
supports
Quoted 6 lines
four modes of operation, configurable via firmware -
Single function mode
Multi function mode
SRIOV mode
MRIOV mode
Quoted 12 lines
Quoted 1 line
The driver patches series will follow this email. This driver has
undergone
Quoted 1 line
significant testing for the past six months in all four modes of
operation,
Quoted 1 line
and is very stable. We would appreciate the community review and
comments
Quoted 6 lines
on this driver.
The modes, besides single function mode, are oriented towards Server I/O virtualization and/or I/O sharing (see PCI SIG SR IOV and MR IOV specs for reference), although they can be used on a single non-virtualized server as well for instance, to run workloads that
would
Quoted 1 line
Quoted 1 line
typically benefit from using separate network cards. In these
scenarios,
Quoted 1 line
Quoted 1 line
X3100 can replace large number of GbE NICs without any system or
network
Quoted 1 line
Quoted 1 line
changes (outside of L2 driver/hardware), while each physical NIC
will be
Quoted 10 lines
Quoted 10 lines
able run up to 10 GbE instead of 1 GbE.
A note on the different modes of operation -
Single-function mode: From Linux stack perspective, the adapter is a typical multi-queue 10GbE pci-e netdev interface (driven by the submitted vxge driver).
Multi-function mode: From Linux stack perspective, the adapter is a multi-function pci-e device where each function is a multi-queue
pci-e
Quoted 2 lines
Quoted 2 lines
netdev interface. This mode has some applications in native Linux environments, but it is primarily designed for use in hypervisors
that
Quoted 1 line
Quoted 1 line
do not yet support SR IOV pci-e extensions. In fact, the
functionality
Quoted 1 line
Quoted 1 line
in this mode is virtually identical to the SR IOV mode described
below,
Quoted 5 lines
Quoted 1 line
except SR IOV support in Linux is not required/expected.
Xen upstream already supports the SR-IOV, and the native Linux and KVM will be supporting it too when 2.6.30 comes out.
Quoted 10 lines
Intel 82576 driver has been patched to enable the SR-IOV capability:
http://patchwork.kernel.org/patch/8063/
http://patchwork.kernel.org/patch/8064/
http://patchwork.kernel.org/patch/8065/
http://patchwork.kernel.org/patch/8066/
Thought one of these patches uses the sysfs interface to receive the NumVFs
from user space, which is deprecated, rest of them still clearly demonstrate
how to convert the traditional PCI NIC driver to the `Physical
Function'
Quoted 1 line
driver by using the SR-IOV API.
Agreed - once SR-IOV support ships in Linux and Xen, using X3100 Multi-function mode becomes optional and the device can/will be used in SR IOV mode. In other hypervisors, transition to SR IOV will take longer time and Multi-function mode will be used for a while.
Enabling SR IOV mode should be transparent to vxge driver - the driver has no SR IOV specific code, and we plan to use the same netdev driver in both Linux and DomU Linux guest. Also (an optional) Xen Dom0 privileged vxge driver stays the same in Multi-function mode and SR IOV mode.
We will look at 82576 patches to understand the changes better, but (at least conceptually :-)) SR-IOV should not require "traditional PCI NIC driver" to change. Some new "knobs" for VF bandwidth allocation, etc. could be optionally added but these are applicable to multi-port or multi-function devices and not SR IOV specific.
The main job of SR IOV support is arguably to translate (reduced) VF PCI config space to full "traditional" PCI space, so networking (or storage or any other subsystem) doesn't know the difference.
What networking resources are implemented behind SR IOV VF is a different question; in x3100 a VF has the same set of NIC resources as a legacy pci function, so a netdev driver can stay the same.
Please let us know if this addresses the comment - alternatively, we can start a different thread since current vxge driver submission doesn't claim SR IOV support. Once SR IOV is supported in the kernel, we will enable SR IOV in x3100 firmware and will test the driver in that mode.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
From vxge driver perspective, each x3100 function is an independent
NIC.
Quoted 1 line
Quoted 1 line
The functions share 10GbE link and pci-e bus, but hardly anything
else
Quoted 2 lines
Quoted 2 lines
inside the ASIC. Features like independent hw reset, statistics, bandwidth/priority allocation and guarantees, etc. are supported.
Also,
Quoted 1 line
Quoted 1 line
typical L2 functions like LSO, interrupt moderation, etc are
supported
Quoted 3 lines
Quoted 3 lines
independently on each function.
In other words, direct hardware access with x3100 doesnctions for
Direct
Quoted 2 lines
Hardware
Quoted 1 line
Access to different Guests or Servers - while keeping
traditional
Quoted 1 line
Quoted 1 line
privileged operations (VLAN membership, etc) under privileged
domain
Quoted 3 lines
Quoted 3 lines
control.
SR IOV mode: Once BIOS and/or OS support SR IOV pci-e extensions,
from
Quoted 1 line
Quoted 1 line
Linux stack perspective the adapter will look like the
multi-function
Quoted 1 line
Quoted 1 line
mode in #2 above. vxge driver will still think that each x3100
function
Quoted 6 lines
Quoted 2 lines
is an independent x3100 pci-e device.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10 GbEPCIe adapter
Quoted 17 lines · Leonid Grossman
Enabling SR IOV mode should be transparent to vxge driver - the driver has no SR IOV specific code, and we plan to use the same netdev driver in both Linux and DomU Linux guest. Also (an optional) Xen Dom0 privileged vxge driver stays the same in Multi-function mode and SR IOV mode.
We will look at 82576 patches to understand the changes better, but (at
least conceptually :-)) SR-IOV should not require "traditional PCI NIC
driver" to change. Some new "knobs" for VF bandwidth allocation, etc.
could be optionally added but these are applicable to multi-port or
multi-function devices and not SR IOV specific.
The main job of SR IOV support is arguably to translate (reduced) VF PCI
config space to full "traditional" PCI space, so networking (or storage
or any other subsystem) doesn't know the difference. What networking resources are implemented behind SR IOV VF is a different question; in x3100 a VF has the same set of NIC resources as a
legacy pci function, so a netdev driver can stay the same.
Quoted 4 lines
Please let us know if this addresses the comment - alternatively, we can start a different thread since current vxge driver submission doesn't claim SR IOV support. Once SR IOV is supported in the kernel, we will enable SR IOV in x3100 firmware and will test the driver in that mode.
For the most part I think the bit you would be interested in is the "sysfs" patch, http://patchwork.kernel.org/patch/8066/, which is what I had used in the original implementation to enable the VFs. I am going to push this to a module parameter similar to your max_config_dev. The rest of the patches handle PF to VF communications and configuration which it sounds like is handled via firmware for your adapter.
Most of the changes you would probably need to make would be in vxge_probe/vxge_remove. All you would end up needing to do is call pci_enable_sriov(pdev, max_config_dev - 1) on your physical function devices and then you would end up getting exactly as many VFs as you need. The call should be safe since I am assuming your VFs don't implement their own SR-IOV capability structures. The cleanup would be pretty strait forward as well since you would just need to call pci_disable_sriov in remove.
Thanks,
Alex
RE: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10 GbEPCIe adapter
Quoted 2 lines
For the most part I think the bit you would be interested in is the "sysfs" patch, http://patchwork.kernel.org/patch/8066/, which is what
I
Quoted 2 lines
had used in the original implementation to enable the VFs. I am going to push this to a module parameter similar to your
max_config_dev.
The
Quoted 2 lines
rest of the patches handle PF to VF communications and configuration which it sounds like is handled via firmware for your adapter.
[Ram]Currently, the messaging interface is not part of this driver submission and each function driver is independent of the other - there are no dependencies between the PF and the VF driver.
Quoted 6 lines
Most of the changes you would probably need to make would be in vxge_probe/vxge_remove. All you would end up needing to do is call
pci_enable_sriov(pdev, max_config_dev - 1)on your physical function devices and then you would end up getting exactly as many VFs as you need. The call should be safe since I am assuming your VFs don't implement their own SR-IOV capability structures. The cleanup would
be
Quoted 2 lines
pretty strait forward as well since you would just need to call
pci_disable_sriovin remove.
[Ram] When the device indicates that it is SRIOV capable in the pci config space, why not create the VF pci config spaces as part of the enumeration process? This way, there would be no change required for the network drivers.
Ram
Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10 GbEPCIe adapter
Quoted 18 lines · Ramkrishna Vepa
Quoted 2 lines
For the most part I think the bit you would be interested in is the "sysfs" patch, http://patchwork.kernel.org/patch/8066/, which is what
I
Quoted 2 lines
had used in the original implementation to enable the VFs. I am going to push this to a module parameter similar to your
max_config_dev.The
Quoted 2 lines
rest of the patches handle PF to VF communications and configuration which it sounds like is handled via firmware for your adapter.
[Ram]Currently, the messaging interface is not part of this driver submission and each function driver is independent of the other - there are no dependencies between the PF and the VF driver.
Quoted 6 lines
Most of the changes you would probably need to make would be in vxge_probe/vxge_remove. All you would end up needing to do is call
pci_enable_sriov(pdev, max_config_dev - 1)on your physical function devices and then you would end up getting exactly as many VFs as you need. The call should be safe since I am assuming your VFs don't implement their own SR-IOV capability structures. The cleanup would
Quoted 7 lines
be
Quoted 2 lines
pretty strait forward as well since you would just need to call
pci_disable_sriovin remove.[Ram] When the device indicates that it is SRIOV capable in the pci config space, why not create the VF pci config spaces as part of the enumeration process? This way, there would be no change required for the network drivers.
Yes, and that's what the PCI subsystem does. If the vxge VF is identical to its PF, then vxge should be able to drive both PF and VF without any modification.
Thanks,
Yu
RE: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10GbEPCIe adapter
Quoted 1 line
Yes, and that's what the PCI subsystem does. If the vxge VF is
identical
Quoted 1 line
to its PF, then vxge should be able to drive both PF and VF without
any
Quoted 1 line
modification.
[Ram] Ok. In that case, is the call to pci_enable/disable_sriov still required for vxge?
Ram
Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series 10GbEPCIe adapter
Quoted 7 lines · Ramkrishna Vepa
Quoted 1 line
Yes, and that's what the PCI subsystem does. If the vxge VF is
identical
Quoted 1 line
to its PF, then vxge should be able to drive both PF and VF without
any
Quoted 1 line
modification.
[Ram] Ok. In that case, is the call to pci_enable/disable_sriov still required for vxge?
Yes, the vxge driver first binds the PF once it's loaded (VF doesn't exist at this time) and calls the SR-IOV API. The VF appears after the SR-IOV is enabled and then the same copy of the vxge driver can bind the VF too if you want to use the VF in the native Linux. Though the hardware is in the SR-IOV mode in this case, it would be equal to the multi-function mode. Or you can assign the VF to the Xen/KVM guest and let another copy of vxge driver (may be vxge for Windows, Solaris, BSD, etc.) running in the guest bind it.
Thanks,
Yu
RE: [ANNOUNCE] New driver vxge for Neterion's X3100 series10GbEPCIe adapter
Quoted 10 lines
-----Original Message-----
From: Yu Zhao [mailto:[email protected]]
Sent: Tuesday, March 31, 2009 10:10 PM
To: Ramkrishna Vepa
Cc: Duyck, Alexander H; Leonid Grossman; Netdev; David Miller
Subject: Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series10GbEPCIe adapter
Quoted 4 lines
Quoted 3 lines · Ramkrishna Vepa
Quoted 1 line
Yes, and that's what the PCI subsystem does. If the vxge VF is
identical
Quoted 1 line
to its PF, then vxge should be able to drive both PF and VF
without
Quoted 3 lines
Quoted 3 lines
any
Quoted 1 line
modification.
[Ram] Ok. In that case, is the call to pci_enable/disable_sriov
still
Quoted 9 lines
Quoted 1 line
required for vxge?
Yes, the vxge driver first binds the PF once it's loaded (VF doesn't exist at this time) and calls the SR-IOV API. The VF appears after the SR-IOV is enabled and then the same copy of the vxge driver can bind the VF too if you want to use the VF in the native Linux. Though the hardware is in the SR-IOV mode in this case, it would be equal to the multi-function mode. Or you can assign the VF to the Xen/KVM guest and let another copy of vxge driver (may be vxge for Windows, Solaris,
BSD,
Quoted 1 line
etc.) running in the guest bind it.
Yu, could you pl. explain why this call is not optional - SR-IOV pci-e code should be able to find SR-IOV capable device and enable all VFs based upon pci-e config space alone, without any help from device-specific PF driver.
Once VFs appear, vxge or any other native netdev driver should be able to bind a VF regardless of PF driver being loaded first (or at all) - there are some use cases that do not assume PF driver presence... Thanks, Leonid
Quoted 3 lines
Thanks,
Yu
Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series10GbEPCIe adapter
Quoted 31 lines · Leonid Grossman
Quoted 10 lines
-----Original Message-----
From: Yu Zhao [mailto:[email protected]]
Sent: Tuesday, March 31, 2009 10:10 PM
To: Ramkrishna Vepa
Cc: Duyck, Alexander H; Leonid Grossman; Netdev; David Miller
Subject: Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series10GbEPCIe adapterQuoted 4 lines
Quoted 3 lines · Ramkrishna Vepa
Quoted 1 line
Yes, and that's what the PCI subsystem does. If the vxge VF is
identical
Quoted 1 line
to its PF, then vxge should be able to drive both PF and VF
without
Quoted 3 lines
Quoted 3 lines
any
Quoted 1 line
modification.
[Ram] Ok. In that case, is the call to pci_enable/disable_sriov
still
Quoted 9 lines
Quoted 1 line
required for vxge?
Yes, the vxge driver first binds the PF once it's loaded (VF doesn't exist at this time) and calls the SR-IOV API. The VF appears after the SR-IOV is enabled and then the same copy of the vxge driver can bind the VF too if you want to use the VF in the native Linux. Though the hardware is in the SR-IOV mode in this case, it would be equal to the multi-function mode. Or you can assign the VF to the Xen/KVM guest and let another copy of vxge driver (may be vxge for Windows, Solaris,
BSD,
Quoted 1 line
etc.) running in the guest bind it.
Quoted 8 lines
Yu, could you pl. explain why this call is not optional - SR-IOV pci-e code should be able to find SR-IOV capable device and enable all VFs based upon pci-e config space alone, without any help from device-specific PF driver.
Once VFs appear, vxge or any other native netdev driver should be able to bind a VF regardless of PF driver being loaded first (or at all) - there are some use cases that do not assume PF driver presence... Thanks, Leonid
The fact is not all drivers will want to enable all of the VFs just because they can. I know in the case of igb I actually prefer to have the default as not enabling all the VFs because as soon as we do the PF itself is no longer multiqueue capable due to the fact that the resources are distributed between the PF and the VFs.
The actual calls themselves are fairly small in terms of enabling things. The pci_enable_sriov call will check before hand if the PCI Config space for SR-IOV exists so it is safe to call in either the VFs or on the PF. Also from our testing internally we prefer to limit the number of VFs allocated to just what is needed since we have seen it can be difficult to sort out PFs and VFs because they can be allocated in such a way that you have to go through with ethtool -i to identify the pci bus/device/function in order to even figure out which port any given interface belongs to.
Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series10GbEPCIe adapter
In reply to Leonid GrossmanQuoted 18 lines · Leonid Grossman
Quoted 4 lines · Ramkrishna Vepa
Quoted 3 lines
Quoted 1 line
Yes, and that's what the PCI subsystem does. If the vxge VF is
identical
Quoted 1 line
to its PF, then vxge should be able to drive both PF and VF
without
Quoted 3 lines
Quoted 3 lines
any
Quoted 1 line
modification.
[Ram] Ok. In that case, is the call to pci_enable/disable_sriov
still
Quoted 9 lines
Quoted 1 line
required for vxge?
Yes, the vxge driver first binds the PF once it's loaded (VF doesn't exist at this time) and calls the SR-IOV API. The VF appears after the SR-IOV is enabled and then the same copy of the vxge driver can bind the VF too if you want to use the VF in the native Linux. Though the hardware is in the SR-IOV mode in this case, it would be equal to the multi-function mode. Or you can assign the VF to the Xen/KVM guest and let another copy of vxge driver (may be vxge for Windows, Solaris,
Quoted 7 lines
BSD,
Quoted 1 line
etc.) running in the guest bind it.
Yu, could you pl. explain why this call is not optional - SR-IOV pci-e code should be able to find SR-IOV capable device and enable all VFs based upon pci-e config space alone, without any help from device-specific PF driver.
Yes, this is true in certain cases. However, there are several things that prevent us to enable the SR-IOV implicitly by the PCI subsystem.
First, the SR-IOV spec says "Once the SRIOV capability is configured enabling VF to be assigned to individual SI, the PF takes on a more supervisory role. For example, the PF can be used to manage device specific functionality such as internal resource allocation to each VF, VF arbitration to shared resources such as the PCIe Link or the Function-specific Link (e.g., a network or storage Link), etc." And some SR-IOV devices follow this suggestion thus their VF cannot work without PF driver pre-exits. Intel 82576 is an example -- it requires the PF driver to allocate tx/rx queues for the VF so the VF can be functional. Only enabling the SR-IOV in the PF PCI config space will end up with VF appearing useless even its PCI config space is visible.
Second, the SR-IOV depends on somethings that are not available before the PCI subsystem is fully initialized. This mean we cannot enable the SR-IOV capability before all PCI device are enumerated. For example, if the VF resides on the bus different than the PF bus, then we can't enable the VF before the bus used by the VF is scan because we don't know if the bus is reserved by the BIOS for the VF or not. Another example is the dependency link used by the PF -- we can't create the sysfs symbolic link indicating the dependency before all PFs in a device are enumerated.
And some SR-IOV devices can support multiple modes at same time. The 82576 can support N VFs + M VMDq modes (N + M = 8), which means sometimes people may want to only enable arbitrary number of VFs. The PCI subsystem can't get value to config the NumVFs unless some one calls the API. I guess only the PF driver can do this.
Quoted 3 lines
Once VFs appear, vxge or any other native netdev driver should be able to bind a VF regardless of PF driver being loaded first (or at all) - there are some use cases that do not assume PF driver presence...
So the PF will not be binded to any driver in these use cases? Can you please elaborate?
Thanks,
Yu
RE: [ANNOUNCE] New driver vxge for Neterion's X3100series10GbEPCIe adapter
Quoted 10 lines
-----Original Message-----
From: Yu Zhao [mailto:[email protected]]
Sent: Tuesday, March 31, 2009 11:56 PM
To: Leonid Grossman
Cc: Ramkrishna Vepa; Duyck, Alexander H; Netdev; David Miller
Subject: Re: [ANNOUNCE] New driver vxge for Neterion's X3100series10GbEPCIe adapter
Quoted 13 lines
Quoted 12 lines · Leonid Grossman
Quoted 4 lines · Ramkrishna Vepa
Quoted 3 lines
Quoted 1 line
Yes, and that's what the PCI subsystem does. If the vxge VF is
identical
Quoted 1 line
to its PF, then vxge should be able to drive both PF and VF
without
Quoted 3 lines
Quoted 3 lines
any
Quoted 1 line
modification.
[Ram] Ok. In that case, is the call to pci_enable/disable_sriov
still
Quoted 3 lines
Quoted 1 line
required for vxge?
Yes, the vxge driver first binds the PF once it's loaded (VF
doesn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
exist at this time) and calls the SR-IOV API. The VF appears after
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
SR-IOV is enabled and then the same copy of the vxge driver can
bind
Quoted 1 line
Quoted 1 line
Quoted 1 line
the VF too if you want to use the VF in the native Linux. Though
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
hardware is in the SR-IOV mode in this case, it would be equal to
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
multi-function mode. Or you can assign the VF to the Xen/KVM guest
and
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
let another copy of vxge driver (may be vxge for Windows, Solaris,
BSD,
Quoted 1 line
etc.) running in the guest bind it.
Yu, could you pl. explain why this call is not optional - SR-IOV
pci-e
Quoted 7 lines
Quoted 3 lines
code should be able to find SR-IOV capable device and enable all VFs based upon pci-e config space alone, without any help from device-specific PF driver.
Yes, this is true in certain cases. However, there are several things that prevent us to enable the SR-IOV implicitly by the PCI subsystem.
Quoted 11 lines
First, the SR-IOV spec says "Once the SRIOV capability is configured enabling VF to be assigned to individual SI, the PF takes on a more supervisory role. For example, the PF can be used to manage device specific functionality such as internal resource allocation to each VF, VF arbitration to shared resources such as the PCIe Link or the Function-specific Link (e.g., a network or storage Link), etc." And some SR-IOV devices follow this suggestion thus their VF cannot work without PF driver pre-exits. Intel 82576 is an example -- it requires the PF driver to allocate tx/rx queues for the VF so the VF can be functional. Only enabling the SR-IOV in the PF PCI config space will end up with VF appearing useless even its PCI config space is visible.
Correct, PF driver can "optionally" manage device specific resources like queue pairs, etc. - this is not the reason though to mandate a PF driver presence in order for VFs to operate.
Arguably PCI code (perhaps with the help of SR PCIM) should be responsible for pci-e resource configuration, and networking driver should be responsible for network resources configuration. If a device like 82576 is implemented in a way that VFs can't operate without a PF driver present, it's a reasonable design trade-off and this case should be supported - but other devices like x3100 do not have this restriction, so PF driver presence should not be a "must have" (pl. see a use case example below).
Quoted 11 lines
Second, the SR-IOV depends on somethings that are not available before the PCI subsystem is fully initialized. This mean we cannot enable the SR-IOV capability before all PCI device are enumerated. For example, if the VF resides on the bus different than the PF bus, then we can't enable the VF before the bus used by the VF is scan because we don't know if the bus is reserved by the BIOS for the VF or not. Another example is the dependency link used by the PF -- we can't create the sysfs symbolic link indicating the dependency before all PFs in a device are enumerated.
Quoted 5 lines
And some SR-IOV devices can support multiple modes at same time. The 82576 can support N VFs + M VMDq modes (N + M = 8), which means sometimes people may want to only enable arbitrary number of VFs. The PCI subsystem can't get value to config the NumVFs unless some one calls the API. I guess only the PF driver can do this.
Have you considered using SR PCIM for this, instead of using a NIC (or another device-specific) driver to configure number of pci functions?
Quoted 2 lines
Quoted 2 lines
Once VFs appear, vxge or any other native netdev driver should be
able
Quoted 1 line
Quoted 1 line
to bind a VF regardless of PF driver being loaded first (or at all)
-
Quoted 4 lines
Quoted 1 line
there are some use cases that do not assume PF driver presence...
So the PF will not be binded to any driver in these use cases? Can you please elaborate?
Yes. For example one of the common use cases for SR IOV is to replace a large number of "legacy" GbE interfaces (as transparently as possible). Assume a customer wants to replace four quad GbE NICs (perhaps running 16 Vlans) with one SR IOV 10GbE card. He considers a PF driver to be a potential security hole and a configuration overhead, and prevents is from loading (perhaps via fw option). His expectation is that 16 VFs will come up and bind to a driver, very much like 16 GbE interfaces did in the original configuration.
So, there should be arguably a way to enable VFs based upon the information in device configuration space alone, without requiring a PF driver to be loaded.
Best, Leonid
Quoted 3 lines
Thanks,
Yu