Skip to contentSkip to conversation

Series short description

PATCH In review Jeff Kirsher 2 participants 4 messages Apr 8, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover Series short description

The following series implements the igbvf driver and modifies the physical function support provided in igb to automatically load a number of Virtual Functions (VFs) equal to the number specified by module parameter num_vfs.

In trying to use the rtnl_link_ops approach we found multiple issues. The SR-IOV interface requires 2 seconds to change number of VFs. As a result doing it one VF at a time requires 14 seconds per device to populate all 7 VFs. The second issue we saw is that when the igbvf driver would attempt to probe the device while we were adding a VF via the rtnl_link_operation and it would deadlock the system unless we released rtnl_lock prior to allocating the VFs.

The original implementation of the igb physical function enabled VFs via a sysfs value. This allowed for specifying which port and the number of VFs to generate. The new approach is a compromise that removes the num_vfs sysfs entry and replaces it to a module parameter similar to what is seen in vxge until we can determine a better way to resolve the issue.

Our testing has found that it is now more difficult to sort out PFs and VFs as they are intermixed during allocation and extra interfaces may be allocated that are not needed since the num_vfs module parameter was defined as a global and not per port. However this is still preferable to the issues seen with the rtnl_link_ops approach.

Notes from testing:
1. Enumeration is inconsistent and vary depending on how the drivers are configured and loaded, for example:
igb=m, igbvf=m
eth0 - 7 are VF
eth8 is PF

The above is on reload, the enumeration is different after initial boot...

igb=y, igbvf=m
eth0 is PF
eth1 - 8 are VFs

2. The inability to easily check which VF belongs to which PF hinders users and the inconsistent enumeration does not make it better. Testing feels that this is a valid concern even in VMs, especially with large number of devices.

---

Alexander Duyck (2):
igb: remove sysfs entry that was used to set the number of vfs
igbvf: add new driver to support 82576 virtual functions

 drivers/net/Kconfig         |   21 
 drivers/net/Makefile        |    1 
 drivers/net/igb/igb_main.c  |  159 +-
 drivers/net/igbvf/Makefile  |   38 +
 drivers/net/igbvf/defines.h |  125 ++
 drivers/net/igbvf/ethtool.c |  540 ++++++++
 drivers/net/igbvf/igbvf.h   |  335 +++++
 drivers/net/igbvf/mbx.c     |  350 +++++
 drivers/net/igbvf/mbx.h     |   75 +
 drivers/net/igbvf/netdev.c  | 2919 +++++++++++++++++++++++++++++++++++++++++++
 drivers/net/igbvf/regs.h    |  108 ++
 drivers/net/igbvf/vf.c      |  398 ++++++
 drivers/net/igbvf/vf.h      |  265 ++++
 13 files changed, 5229 insertions(+), 105 deletions(-)
 create mode 100644 drivers/net/igbvf/Makefile
 create mode 100644 drivers/net/igbvf/defines.h
 create mode 100644 drivers/net/igbvf/ethtool.c
 create mode 100644 drivers/net/igbvf/igbvf.h
 create mode 100644 drivers/net/igbvf/mbx.c
 create mode 100644 drivers/net/igbvf/mbx.h
 create mode 100644 drivers/net/igbvf/netdev.c
 create mode 100644 drivers/net/igbvf/regs.h
 create mode 100644 drivers/net/igbvf/vf.c
 create mode 100644 drivers/net/igbvf/vf.h
Signature · 2 lines
Cheers,
Jeff
#1 lore raw

PATCH v4 1/2 igbvf: add new driver to support 82576 virtual functions

From: Alexander Duyck <[email protected]>

This adds an igbvf driver to handle virtual functions provided by the igb driver when SR-IOV has been enabled. A virtual function is a lightweight pci-e function that supports a single queue and shares resources with the 82576 physical function contained within the igb driver.

To spawn virtual functions from the igb driver all that is needed is to enable CONFIG_PCI_IOV and have an 82576 Ethernet adapter on a system that supports SR-IOV in the BIOS. The virtual functions will appear after the interface is loaded.

---

 drivers/net/Kconfig         |   21 
 drivers/net/Makefile        |    1 
 drivers/net/igbvf/Makefile  |   38 +
 drivers/net/igbvf/defines.h |  125 ++
 drivers/net/igbvf/ethtool.c |  540 ++++++++
 drivers/net/igbvf/igbvf.h   |  335 +++++
 drivers/net/igbvf/mbx.c     |  350 +++++
 drivers/net/igbvf/mbx.h     |   75 +
 drivers/net/igbvf/netdev.c  | 2919 +++++++++++++++++++++++++++++++++++++++++++
 drivers/net/igbvf/regs.h    |  108 ++
 drivers/net/igbvf/vf.c      |  398 ++++++
 drivers/net/igbvf/vf.h      |  265 ++++
 12 files changed, 5175 insertions(+), 0 deletions(-)
 create mode 100644 drivers/net/igbvf/Makefile
 create mode 100644 drivers/net/igbvf/defines.h
 create mode 100644 drivers/net/igbvf/ethtool.c
 create mode 100644 drivers/net/igbvf/igbvf.h
 create mode 100644 drivers/net/igbvf/mbx.c
 create mode 100644 drivers/net/igbvf/mbx.h
 create mode 100644 drivers/net/igbvf/netdev.c
 create mode 100644 drivers/net/igbvf/regs.h
 create mode 100644 drivers/net/igbvf/vf.c
 create mode 100644 drivers/net/igbvf/vf.h
drivers/net/Kconfig +21 -0
index 9e7baec..17ae9e4 100644
@@ -2056,6 +2056,27 @@ config IGB_DCA
 	  driver.  DCA is a method for warming the CPU cache before data
 	  is used, with the intent of lessening the impact of cache misses.
 
+config IGBVF
+       tristate "Intel(R) 82576 Virtual Function Ethernet support"
+       depends on PCI
+       ---help---
+         This driver supports Intel(R) 82576 virtual functions.  For more
+         information on how to identify your adapter, go to the Adapter &
+         Driver ID Guide at:
+
+         <http://support.intel.com/support/network/adapter/pro100/21397.htm>
+
+         For general information and support, go to the Intel support
+         website at:
+
+         <http://support.intel.com>
+
+         More specific information on configuring the driver is in
+         <file:Documentation/networking/e1000.txt>.
+
+         To compile this driver as a module, choose M here. The module
+         will be called igbvf.
+
 source "drivers/net/ixp2000/Kconfig"
 
 config MYRI_SBUS
drivers/net/Makefile +1 -0
index edc9a0d..1fc4602 100644
@@ -6,6 +6,7 @@ obj-$(CONFIG_E1000) += e1000/
 obj-$(CONFIG_E1000E) += e1000e/
 obj-$(CONFIG_IBM_NEW_EMAC) += ibm_newemac/
 obj-$(CONFIG_IGB) += igb/
+obj-$(CONFIG_IGBVF) += igbvf/
 obj-$(CONFIG_IXGBE) += ixgbe/
 obj-$(CONFIG_IXGB) += ixgb/
 obj-$(CONFIG_IP1000) += ipg.o
drivers/net/igbvf/Makefile new file +38 -0
new file mode 100644
index 0000000..c2f150d
@@ -0,0 +1,38 @@
+################################################################################
+#
+# Intel(R) 82576 Virtual Function Linux driver
+# Copyright(c) 2009 Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# The full GNU General Public License is included in this distribution in
+# the file called "COPYING".
+#
+# Contact Information:
+# e1000-devel Mailing List <[email protected]>
+# Intel Corporation, 5200 N.E. Elam Young Parkway, Hillsboro, OR 97124-6497
+#
+################################################################################
+
+#
+# Makefile for the Intel(R) 82576 VF ethernet driver
+#
+
+obj-$(CONFIG_IGBVF) += igbvf.o
+
+igbvf-objs := vf.o \
+              mbx.o \
+              ethtool.o \
+              netdev.o
+
drivers/net/igbvf/defines.h new file +125 -0

125 lines not on this page · raw message

drivers/net/igbvf/ethtool.c new file +540 -0

540 lines not on this page · raw message

drivers/net/igbvf/igbvf.h new file +335 -0

335 lines not on this page · raw message

drivers/net/igbvf/mbx.c new file +350 -0

350 lines not on this page · raw message

drivers/net/igbvf/mbx.h new file +75 -0

75 lines not on this page · raw message

drivers/net/igbvf/netdev.c new file +2919 -0

1,503 lines not on this page · raw message

drivers/net/igbvf/regs.h new file +108 -0

108 lines not on this page · raw message

drivers/net/igbvf/vf.c new file +398 -0

398 lines not on this page · raw message

drivers/net/igbvf/vf.h new file +265 -0

265 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#2 lore raw

PATCH v4 2/2 igb: remove sysfs entry that was used to set the number of vfs

In reply to Jeff Kirsher

From: Alexander Duyck <[email protected]>

This patch removes the sysfs entry num_vfs which was added to support enabling pci virtual functions for 82576.

To prevent VFs from loading automatically a module parameter "max_vfs" was added so that the number of VFs per PF can be limited. This is especially useful when 4 or more 82576 ports are on the system because otherwise to load all VFs would result in 8 interface per physical port.

---

 drivers/net/igb/igb_main.c |  159 +++++++++++++++-----------------------------
 1 files changed, 54 insertions(+), 105 deletions(-)
drivers/net/igb/igb_main.c +54 -105
index 6b0697c..db7274e 100644
@@ -152,14 +152,13 @@ static struct notifier_block dca_notifier = {
 /* for netdump / net console */
 static void igb_netpoll(struct net_device *);
 #endif
-
 #ifdef CONFIG_PCI_IOV
-static ssize_t igb_set_num_vfs(struct device *, struct device_attribute *,
-                               const char *, size_t);
-static ssize_t igb_show_num_vfs(struct device *, struct device_attribute *,
-                               char *);
-DEVICE_ATTR(num_vfs, S_IRUGO | S_IWUSR, igb_show_num_vfs, igb_set_num_vfs);
-#endif
+static unsigned int max_vfs = 0;
+module_param(max_vfs, uint, 0);
+MODULE_PARM_DESC(max_vfs, "Maximum number of virtual functions to allocate "
+                 "per physical function");
+#endif /* CONFIG_PCI_IOV */
+
 static pci_ers_result_t igb_io_error_detected(struct pci_dev *,
 		     pci_channel_state_t);
 static pci_ers_result_t igb_io_slot_reset(struct pci_dev *);
@@ -671,6 +670,21 @@ static void igb_set_interrupt_capability(struct igb_adapter *adapter)
 
 	/* If we can't do MSI-X, try MSI */
 msi_only:
+#ifdef CONFIG_PCI_IOV
+	/* disable SR-IOV for non MSI-X configurations */
+	if (adapter->vf_data) {
+		struct e1000_hw *hw = &adapter->hw;
+		/* disable iov and allow time for transactions to clear */
+		pci_disable_sriov(adapter->pdev);
+		msleep(500);
+
+		kfree(adapter->vf_data);
+		adapter->vf_data = NULL;
+		wr32(E1000_IOVCTL, E1000_IOVCTL_REUSE_VFQ);
+		msleep(100);
+		dev_info(&adapter->pdev->dev, "IOV Disabled\n");
+	}
+#endif
 	adapter->num_rx_queues = 1;
 	adapter->num_tx_queues = 1;
 	if (!pci_enable_msi(adapter->pdev))
@@ -1238,6 +1252,39 @@ static int __devinit igb_probe(struct pci_dev *pdev,
 	if (err)
 		goto err_sw_init;
 
+#ifdef CONFIG_PCI_IOV
+	/* since iov functionality isn't critical to base device function we
+	 * can accept failure.  If it fails we don't allow iov to be enabled */
+	if (hw->mac.type == e1000_82576) {
+		/* 82576 supports a maximum of 7 VFs in addition to the PF */
+		unsigned int num_vfs = (max_vfs > 7) ? 7 : max_vfs;
+		int i;
+		unsigned char mac_addr[ETH_ALEN];
+
+		if (num_vfs)
+			adapter->vf_data = kcalloc(num_vfs,
+						sizeof(struct vf_data_storage),
+						GFP_KERNEL);
+		if (!adapter->vf_data) {
+			dev_err(&pdev->dev, "Could not allocate VF private "
+				"data - IOV enable failed\n");
+		} else {
+			err = pci_enable_sriov(pdev, num_vfs);
+			if (!err) {
+				adapter->vfs_allocated_count = num_vfs;
+				dev_info(&pdev->dev, "%d vfs allocated\n", num_vfs);
+				for (i = 0; i < adapter->vfs_allocated_count; i++) {
+					random_ether_addr(mac_addr);
+					igb_set_vf_mac(adapter, i, mac_addr);
+				}
+			} else {
+				kfree(adapter->vf_data);
+				adapter->vf_data = NULL;
+			}
+		}
+	}
+
+#endif
 	/* setup the private structure */
 	err = igb_sw_init(adapter);
 	if (err)
@@ -1397,19 +1444,6 @@ static int __devinit igb_probe(struct pci_dev *pdev,
 	if (err)
 		goto err_register;
 
-#ifdef CONFIG_PCI_IOV
-	/* since iov functionality isn't critical to base device function we
-	 * can accept failure.  If it fails we don't allow iov to be enabled */
-	if (hw->mac.type == e1000_82576) {
-		err = pci_enable_sriov(pdev, 0);
-		if (!err)
-			err = device_create_file(&netdev->dev,
-			                         &dev_attr_num_vfs);
-		if (err)
-			dev_err(&pdev->dev, "Failed to initialize IOV\n");
-	}
-
-#endif
 #ifdef CONFIG_IGB_DCA
 	if (dca_add_requester(&pdev->dev) == 0) {
 		adapter->flags |= IGB_FLAG_DCA_ENABLED;
@@ -5422,89 +5456,4 @@ static void igb_vmm_control(struct igb_adapter *adapter)
 	igb_vmdq_set_replication_pf(hw, true);
 }
 
-#ifdef CONFIG_PCI_IOV
-static ssize_t igb_show_num_vfs(struct device *dev,
-                                struct device_attribute *attr, char *buf)
-{
-	struct igb_adapter *adapter = netdev_priv(to_net_dev(dev));
-
-	return sprintf(buf, "%d\n", adapter->vfs_allocated_count);
-}
-
-static ssize_t igb_set_num_vfs(struct device *dev,
-                               struct device_attribute *attr,
-                               const char *buf, size_t count)
-{
-	struct net_device *netdev = to_net_dev(dev);
-	struct igb_adapter *adapter = netdev_priv(netdev);
-	struct e1000_hw *hw = &adapter->hw;
-	struct pci_dev *pdev = adapter->pdev;
-	unsigned int num_vfs, i;
-	unsigned char mac_addr[ETH_ALEN];
-	int err;
-
-	sscanf(buf, "%u", &num_vfs);
-
-	if (num_vfs > 7)
-		num_vfs = 7;
-
-	/* value unchanged do nothing */
-	if (num_vfs == adapter->vfs_allocated_count)
-		return count;
-
-	if (netdev->flags & IFF_UP)
-		igb_close(netdev);
-
-	igb_reset_interrupt_capability(adapter);
-	igb_free_queues(adapter);
-	adapter->tx_ring = NULL;
-	adapter->rx_ring = NULL;
-	adapter->vfs_allocated_count = 0;
-
-	/* reclaim resources allocated to VFs since we are changing count */
-	if (adapter->vf_data) {
-		/* disable iov and allow time for transactions to clear */
-		pci_disable_sriov(pdev);
-		msleep(500);
-
-		kfree(adapter->vf_data);
-		adapter->vf_data = NULL;
-		wr32(E1000_IOVCTL, E1000_IOVCTL_REUSE_VFQ);
-		msleep(100);
-		dev_info(&pdev->dev, "IOV Disabled\n");
-	}
-
-	if (num_vfs) {
-		adapter->vf_data = kcalloc(num_vfs,
-		                           sizeof(struct vf_data_storage),
-		                           GFP_KERNEL);
-		if (!adapter->vf_data) {
-			dev_err(&pdev->dev, "Could not allocate VF private "
-				"data - IOV enable failed\n");
-		} else {
-			err = pci_enable_sriov(pdev, num_vfs);
-			if (!err) {
-				adapter->vfs_allocated_count = num_vfs;
-				dev_info(&pdev->dev, "%d vfs allocated\n", num_vfs);
-				for (i = 0; i < adapter->vfs_allocated_count; i++) {
-					random_ether_addr(mac_addr);
-					igb_set_vf_mac(adapter, i, mac_addr);
-				}
-			} else {
-				kfree(adapter->vf_data);
-				adapter->vf_data = NULL;
-			}
-		}
-	}
-
-	igb_set_interrupt_capability(adapter);
-	igb_alloc_queues(adapter);
-	igb_reset(adapter);
-
-	if (netdev->flags & IFF_UP)
-		igb_open(netdev);
-
-	return count;
-}
-#endif /* CONFIG_PCI_IOV */
 /* igb_main.c */
maintainer
#3 lore raw
In reply to Jeff Kirsher

From: Jeff Kirsher <[email protected]>
Date: Tue, 07 Apr 2009 17:37:20 -0700

Quoted 3 lines

Alexander Duyck (2):
igb: remove sysfs entry that was used to set the number of vfs
igbvf: add new driver to support 82576 virtual functions

Both applied, thanks.