Skip to contentSkip to conversation

igbvf: add new driver to support 82576 virtual functions

PATCH Superseded Jeff Kirsher 8 participants 17 messages Mar 25, 2009 – Mar 27, 2009

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 igbvf: add new driver to support 82576 virtual functions

From: Alexander Duyck <[email protected]>

This adds an igbvf driver to handle virtual functions provided by the igb driver when SR-IOV has been enabled. A virtual function is a lightweight pci-e function that supports a single queue and shares resources with the 82576 physical function contained within the igb driver.

To spawn virtual functions from the igb driver all that is needed is to issue a echo X > /sys/class/ethY/num_vfs. X can be a value between 0 and 7, 0 will disable SR-IOV functionality for the port and is the default. If the num_vfs sysfs entry is not present then the device does not have SR-IOV capability enabled either in software or hardware.

---

 drivers/net/Kconfig         |   21 
 drivers/net/Makefile        |    1 
 drivers/net/igbvf/Makefile  |   38 +
 drivers/net/igbvf/defines.h |  125 ++
 drivers/net/igbvf/ethtool.c |  540 ++++++++
 drivers/net/igbvf/igbvf.h   |  335 +++++
 drivers/net/igbvf/mbx.c     |  350 +++++
 drivers/net/igbvf/mbx.h     |   75 +
 drivers/net/igbvf/netdev.c  | 2917 +++++++++++++++++++++++++++++++++++++++++++
 drivers/net/igbvf/regs.h    |  108 ++
 drivers/net/igbvf/vf.c      |  398 ++++++
 drivers/net/igbvf/vf.h      |  265 ++++
 12 files changed, 5173 insertions(+), 0 deletions(-)
 create mode 100644 drivers/net/igbvf/Makefile
 create mode 100644 drivers/net/igbvf/defines.h
 create mode 100644 drivers/net/igbvf/ethtool.c
 create mode 100644 drivers/net/igbvf/igbvf.h
 create mode 100644 drivers/net/igbvf/mbx.c
 create mode 100644 drivers/net/igbvf/mbx.h
 create mode 100644 drivers/net/igbvf/netdev.c
 create mode 100644 drivers/net/igbvf/regs.h
 create mode 100644 drivers/net/igbvf/vf.c
 create mode 100644 drivers/net/igbvf/vf.h
drivers/net/Kconfig +21 -0
index e5ffc1c..1f3d29d 100644
@@ -2048,6 +2048,27 @@ config IGB_DCA
 	  driver.  DCA is a method for warming the CPU cache before data
 	  is used, with the intent of lessening the impact of cache misses.
 
+config IGBVF
+       tristate "Intel(R) 82576 Virtual Function Ethernet support"
+       depends on PCI
+       ---help---
+         This driver supports Intel(R) 82576 virtual functions.  For more
+         information on how to identify your adapter, go to the Adapter &
+         Driver ID Guide at:
+
+         <http://support.intel.com/support/network/adapter/pro100/21397.htm>
+
+         For general information and support, go to the Intel support
+         website at:
+
+         <http://support.intel.com>
+
+         More specific information on configuring the driver is in
+         <file:Documentation/networking/e1000.txt>.
+
+         To compile this driver as a module, choose M here. The module
+         will be called igbvf.
+
 source "drivers/net/ixp2000/Kconfig"
 
 config MYRI_SBUS
drivers/net/Makefile +1 -0
index 758ecdf..43c8510 100644
@@ -6,6 +6,7 @@ obj-$(CONFIG_E1000) += e1000/
 obj-$(CONFIG_E1000E) += e1000e/
 obj-$(CONFIG_IBM_NEW_EMAC) += ibm_newemac/
 obj-$(CONFIG_IGB) += igb/
+obj-$(CONFIG_IGBVF) += igbvf/
 obj-$(CONFIG_IXGBE) += ixgbe/
 obj-$(CONFIG_IXGB) += ixgb/
 obj-$(CONFIG_IP1000) += ipg.o
drivers/net/igbvf/Makefile new file +38 -0
new file mode 100644
index 0000000..c2f150d
@@ -0,0 +1,38 @@
+################################################################################
+#
+# Intel(R) 82576 Virtual Function Linux driver
+# Copyright(c) 2009 Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# The full GNU General Public License is included in this distribution in
+# the file called "COPYING".
+#
+# Contact Information:
+# e1000-devel Mailing List <[email protected]>
+# Intel Corporation, 5200 N.E. Elam Young Parkway, Hillsboro, OR 97124-6497
+#
+################################################################################
+
+#
+# Makefile for the Intel(R) 82576 VF ethernet driver
+#
+
+obj-$(CONFIG_IGBVF) += igbvf.o
+
+igbvf-objs := vf.o \
+              mbx.o \
+              ethtool.o \
+              netdev.o
+
drivers/net/igbvf/defines.h new file +125 -0

125 lines not on this page · raw message

drivers/net/igbvf/ethtool.c new file +540 -0

540 lines not on this page · raw message

drivers/net/igbvf/igbvf.h new file +335 -0

335 lines not on this page · raw message

drivers/net/igbvf/mbx.c new file +350 -0

350 lines not on this page · raw message

drivers/net/igbvf/mbx.h new file +75 -0

75 lines not on this page · raw message

drivers/net/igbvf/netdev.c new file +2917 -0

1,503 lines not on this page · raw message

drivers/net/igbvf/regs.h new file +108 -0

108 lines not on this page · raw message

drivers/net/igbvf/vf.c new file +398 -0

398 lines not on this page · raw message

drivers/net/igbvf/vf.h new file +265 -0

265 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 14 lines · Jeff Kirsher

From: Alexander Duyck <[email protected]>

This adds an igbvf driver to handle virtual functions provided by the igb driver when SR-IOV has been enabled.  A virtual function is a lightweight pci-e function that supports a single queue and shares resources with the 82576 physical function contained within the igb driver.

To spawn virtual functions from the igb driver all that is needed is to issue a echo X > /sys/class/ethY/num_vfs.  X can be a value between 0 and 7, 0 will disable SR-IOV functionality for the port and is the default.  If the num_vfs sysfs entry is not present then the device does not have SR-IOV capability enabled either in software or hardware.

Quoted 3 lines

---

Only difference between v2 and v3 is the addition of the following line

Quoted 1 line

+#include <linux/vmalloc.h>

into drivers/net/igbvf/ethtool.c. We verified that this fixes the compilation issues on PPC.

Signature · 2 lines
Cheers,
Jeff
#2 lore raw
In reply to Jeff Kirsher
Quoted 8 lines · On Wed, 25 Mar 2009 14:52:48 -0700 Jeff Kirsher <[email protected]>…

From: Alexander Duyck <[email protected]>

This adds an igbvf driver to handle virtual functions provided by the igb driver when SR-IOV has been enabled. A virtual function is a lightweight pci-e function that supports a single queue and shares resources with the 82576 physical function contained within the igb driver.

Quoted 8 lines

To spawn virtual functions from the igb driver all that is needed is to issue a echo X > /sys/class/ethY/num_vfs. X can be a value between 0 and 7, 0 will disable SR-IOV functionality for the port and is the default. If the num_vfs sysfs entry is not present then the device does not have SR-IOV capability enabled either in software or hardware.

Please don't use sysfs for this.
Instead build a proper rtnl_link netlink interface (see macvlan).

Intel doesn't invent unique hardware, other vendors will do the same thing (or follow your lead), so doing it right the first time for these kind of devices makes sense.

#3 lore raw
Quoted 20 lines · Stephen Hemminger
Quoted 8 lines · On Wed, 25 Mar 2009 14:52:48 -0700 Jeff Kirsher <[email protected]>…

From: Alexander Duyck <[email protected]>

This adds an igbvf driver to handle virtual functions provided by the igb driver when SR-IOV has been enabled. A virtual function is a lightweight pci-e function that supports a single queue and shares resources with the 82576 physical function contained within the igb driver.

Quoted 8 lines

To spawn virtual functions from the igb driver all that is needed is to issue a echo X > /sys/class/ethY/num_vfs. X can be a value between 0 and 7, 0 will disable SR-IOV functionality for the port and is the default. If the num_vfs sysfs entry is not present then the device does not have SR-IOV capability enabled either in software or hardware.

Quoted 6 lines

Please don't use sysfs for this.
Instead build a proper rtnl_link netlink interface (see macvlan).

Intel doesn't invent unique hardware, other vendors will do the same thing (or follow your lead), so doing it right the first time for these kind of devices makes sense.

The sysfs part of this is already in net-next as it isn't part of the igbvf driver it is part of igb. It was added in over a month ago:
http://patchwork.ozlabs.org/patch/23471/

I will see what we can do to make use of a netlink interface. I'm honestly not even sure if it will work well for this kind of setup since the VF driver won't even be running on the same OS in most cases. For now the sysfs code in igb is actually quite minimal, and the main goal was to keep the interface as simple as possible which is what I have done.

We are already planning to make a number of changes to support Host configuration of the guest interfaces for 2.6.31, and a number of those included sysfs entries that I had rejected for this release. What I can probably do is see about trying to make all of those changes be pushed over to a netlink interface since it would make much more sense when we have multiple items to configure other than just setting the number of VFs.

Thanks,

Alex

maintainer
#4 lore raw

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 15:33:31 -0700

Quoted 10 lines

The sysfs part of this is already in net-next as it isn't part of the igbvf driver it is part of igb. It was added in over a month
ago: http://patchwork.ozlabs.org/patch/23471/

I will see what we can do to make use of a netlink interface. I'm honestly not even sure if it will work well for this kind of setup since the VF driver won't even be running on the same OS in most cases. For now the sysfs code in igb is actually quite minimal, and the main goal was to keep the interface as simple as possible which is what I have done.

I have to agree with Stephen that the way to create new links of virtual and similar devices is to use rtnl_link.

We have very good infrastructure for this, and if I understand things correctly iproute2 might even work with an igb using rtnl_link with zero modifications to the tool sources.

Please fix this up, it's a very reasonable request for such a large new piece of driver infrastructure. And since this is a new driver, you have no time constraints for submission either.

#5 lore raw
Quoted 17 lines · David Miller

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 15:33:31 -0700

Quoted 10 lines

The sysfs part of this is already in net-next as it isn't part of the igbvf driver it is part of igb. It was added in over a month
ago: http://patchwork.ozlabs.org/patch/23471/

I will see what we can do to make use of a netlink interface. I'm honestly not even sure if it will work well for this kind of setup since the VF driver won't even be running on the same OS in most cases. For now the sysfs code in igb is actually quite minimal, and the main goal was to keep the interface as simple as possible which is what I have done.

I have to agree with Stephen that the way to create new links of virtual and similar devices is to use rtnl_link.

Quoted 8 lines

We have very good infrastructure for this, and if I understand things correctly iproute2 might even work with an igb using rtnl_link with zero modifications to the tool sources.

Please fix this up, it's a very reasonable request for such a large new piece of driver infrastructure. And since this is a new driver, you have no time constraints for submission either.

Thanks for allowing me more time, but the issue isn't the igbvf driver.
The part that Stephen is referencing will be an issue in the igb
driver. That is why I pointed him to http://patchwork.ozlabs.org/patch/23471/ since it is the part that contains the code that he has issues with.

Since the issue isn't the igbvf driver there is no reason for it to be held up. If you want to revert the offending patch feel free to and what I will do is send a patch out that just statically set the number of VFs to 7 if the kernel and device supports enabling SR-IOV. I didn't want to take that approach since it means that it disables multi-queue for igb on those devices but I feel like I have been backed into a corner on this and don't have many other options.

Thanks,

Alex

#6 lore raw
Quoted 33 lines · Alexander Duyck
Quoted 17 lines · David Miller

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 15:33:31 -0700

Quoted 10 lines

The sysfs part of this is already in net-next as it isn't part of the igbvf driver it is part of igb. It was added in over a month
ago: http://patchwork.ozlabs.org/patch/23471/

I will see what we can do to make use of a netlink interface. I'm honestly not even sure if it will work well for this kind of setup since the VF driver won't even be running on the same OS in most cases. For now the sysfs code in igb is actually quite minimal, and the main goal was to keep the interface as simple as possible which is what I have done.

I have to agree with Stephen that the way to create new links of virtual and similar devices is to use rtnl_link.

Quoted 8 lines

We have very good infrastructure for this, and if I understand things correctly iproute2 might even work with an igb using rtnl_link with zero modifications to the tool sources.

Please fix this up, it's a very reasonable request for such a large new piece of driver infrastructure. And since this is a new driver, you have no time constraints for submission either.

Thanks for allowing me more time, but the issue isn't the igbvf driver.
The part that Stephen is referencing will be an issue in the igb
driver. That is why I pointed him to
http://patchwork.ozlabs.org/patch/23471/ since it is the part that contains the code that he has issues with.

Quoted 7 lines

Since the issue isn't the igbvf driver there is no reason for it to be held up. If you want to revert the offending patch feel free to and what I will do is send a patch out that just statically set the number of VFs to 7 if the kernel and device supports enabling SR-IOV. I didn't want to take that approach since it means that it disables multi-queue for igb on those devices but I feel like I have been backed into a corner on this and don't have many other options.

We do have some powerful tools for network device configuration, but they still can't cover the multi queue and SR-IOV NICs (also David Woodhouse's Solos DSL, etc.) configuration requirement. Unless someone improves the existing tools to support configuring these new hardware features, people would keep trying to use the sysfs which is the easiest way so far.

Thanks,
Yu

maintainer
#9 lore raw
In reply to Alexander Duyck

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 17:54:55 -0700

Quoted 2 lines

Since the issue isn't the igbvf driver there is no reason for it to be held up.

I disagree, I think both cases should be fixed.

Just because we do something already never means that it's ok to proliferate the mistake further.

maintainer
#7 lore raw
In reply to Yu Zhao

From: Yu Zhao <[email protected]>
Date: Thu, 26 Mar 2009 10:33:43 +0800

Quoted 3 lines

We do have some powerful tools for network device configuration, but they still can't cover the multi queue and SR-IOV NICs (also David Woodhouse's Solos DSL, etc.) configuration requirement.

We have a generic link creation mechanism in rtnl_link so I think this claim wrt. SR-IOV is false.

Believe it or not, somebody took the time to make a generic mechanism for things many virtual network device drivers have to do, and igb/igbvf can use it too.

#10 lore raw
In reply to David Miller
Quoted 7 lines
Quoted 2 lines

Since the issue isn't the igbvf driver there is no reason for it to be held up.

I disagree, I think both cases should be fixed.

Just because we do something already never means that it's ok to proliferate the mistake further.

The igbvf driver doesn't proliferate any mistake. The num_vfs sysfs attribute that everyone finds so objectionable is in the igb driver, and if you follow the patchwork URL that was provided, you would see that you applied the patch adding it back in February. The only thing the igbvf driver does relating to this is bind to the PCI ID of the virtual functions created by the igb driver.

- R.

#12 lore raw
In reply to David Miller
Quoted 10 lines · David Miller

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 17:54:55 -0700

Quoted 2 lines

Since the issue isn't the igbvf driver there is no reason for it to be held up.

I disagree, I think both cases should be fixed.

Just because we do something already never means that it's ok to proliferate the mistake further.

That isn't what I mean. The code he is referring to exists nowhere in the igbvf driver. I suppose I can edit the igbvf commit comments so that they don't mention the sysfs entry, but the code is in the igb driver.

Also I just want to clarify. It isn't fair to compare igbvf to macvlan. A better comparison would be virtio since it is meant to run on a guest, not on the hypervisor/DOM0 OS. I have also been looking all over for an example that is even close to what we are looking for and the sad fact is that the closest thing I can find in the kernel is the DCB netlink code since it effects the number of queues in use. , and I don't have the time to implement anything like that before the merge window is closed.

What I want to know is if I get rid of the sysfs entry in igb and go with a static number of VFs, and remove the commit comment referencing the sysfs entry for igbvf will that be enough to get this accepted for 2.6.30?

Thanks,

Alex

maintainer
#11 lore raw
In reply to Roland Dreier

From: Roland Dreier <[email protected]>
Date: Wed, 25 Mar 2009 20:21:08 -0700

Quoted 6 lines

The igbvf driver doesn't proliferate any mistake. The num_vfs sysfs attribute that everyone finds so objectionable is in the igb driver, and if you follow the patchwork URL that was provided, you would see that you applied the patch adding it back in February. The only thing the igbvf driver does relating to this is bind to the PCI ID of the virtual functions created by the igb driver.

I am saying I made a mistake when merging that.

Can we continue forward now?

Thanks.

maintainer
#13 lore raw
In reply to Alexander Duyck

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 20:27:28 -0700

Quoted 9 lines · David Miller
Quoted 8 lines

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 17:54:55 -0700

Quoted 2 lines

Since the issue isn't the igbvf driver there is no reason for it to be held up.

I disagree, I think both cases should be fixed.

Quoted 7 lines
Quoted 2 lines

Just because we do something already never means that it's ok to proliferate the mistake further.

That isn't what I mean. The code he is referring to exists nowhere in the igbvf driver. I suppose I can edit the igbvf commit comments so that they don't mention the sysfs entry, but the code is in the igb driver.

I know it's in the igb driver, I fully understand that.

And I'm saying it was a mistake to merge that, it slipped past me in my review of that change, and we need to move forward and get rid of this thing.

#8 lore raw
In reply to Yu Zhao
Quoted 5 lines · Yu Zhao

We do have some powerful tools for network device configuration, but they still can't cover the multi queue and SR-IOV NICs (also David Woodhouse's Solos DSL, etc.) configuration requirement. Unless someone improves the existing tools to support configuring these new hardware features, people would keep trying to use the sysfs which is the easiest way so far.

So what are these missing features in the existing interfaces?

#14 lore raw

RFC 1/1 igb: first draft of igb rtnl_link_ops interface for vf creation (was Re: [net-next PATCH v3] igbvf: add new driver to support 82576 virtual functions)

In reply to David Miller
Quoted 18 lines · David Miller

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 20:27:28 -0700

Quoted 6 lines · David Miller
Quoted 5 lines

From: Alexander Duyck <[email protected]>
Date: Wed, 25 Mar 2009 17:54:55 -0700

Quoted 2 lines

Since the issue isn't the igbvf driver there is no reason for it to be held up.

Quoted 8 lines
Quoted 4 lines

I disagree, I think both cases should be fixed.

Just because we do something already never means that it's ok to proliferate the mistake further.

That isn't what I mean. The code he is referring to exists nowhere in the igbvf driver. I suppose I can edit the igbvf commit comments so that they don't mention the sysfs entry, but the code is in the igb driver.

Quoted 5 lines

I know it's in the igb driver, I fully understand that.

And I'm saying it was a mistake to merge that, it slipped past me in my review of that change, and we need to move forward and get rid of this thing.

Ok, I have already submitted the patches to remove the sysfs entry, and the comments about it from the commit in igbvf to Jeff. Hopefully we can get the pushed through sometime soon.

In the meantime I have been working on the rtnl_link_ops approach and I think I have a few things going but I wanted to get some input before I go much further.

First, is it ok for me to call rtnl_unlock prior to doing my settings changes on the sriov config space, followed by rtnl_lock afterwards in my newlink and dellink operations? I ask because I had to do this in order to prevent a deadlock when the pci-hotplug events fired for the vfs and called unregister/register_netdev.

Second is it acceptable for me to just free the netdev at the end of newlink and call delete on the PF interface directly? I ask because I don't have any use for the netdevs that are generated and I cannot call delete on specific VFs anyway since they are allocated/freed in LIFO order so I would always have to free the last one I allocated.

I have included a patch for review below that implements these changes against the current driver. Please feel free to comment.

Thanks,

Alex

--

This patch converts igb from using a sysfs interface to allocate vfs to using the rtnl_link interface.

Basically the new interface work like this to add a VF:
ip link add eth3 type igbpf

To delete a VF it would be:
ip link del eth3 type igbpf

---

drivers/net/igb/igb_main.c | 120 +++++++++++++++++++++++++++++---------------

1 files changed, 80 insertions(+), 40 deletions(-)
drivers/net/igb/igb_main.c +0 -0
index 7d3ac76..d1e9425 100644
@@ -154,11 +154,17 @@ static void igb_netpoll(struct net_device *);

#endif

#ifdef CONFIG_PCI_IOV
-static ssize_t igb_set_num_vfs(struct device *, struct device_attribute *,
- const char *, size_t);
-static ssize_t igb_show_num_vfs(struct device *, struct device_attribute *,
- char *);
-DEVICE_ATTR(num_vfs, S_IRUGO | S_IWUSR, igb_show_num_vfs, igb_set_num_vfs); +static int igb_new_vf(struct net_device *dev, struct nlattr *tb[],
+ struct nlattr *data[]);
+static void igb_del_vf(struct net_device *dev);
+static void igb_setup_vf(struct net_device *dev);
+
+static struct rtnl_link_ops igbpf_link_ops __read_mostly = {
+ .kind = "igbpf",
+ .setup = igb_setup_vf,
+ .newlink = igb_new_vf,
+ .dellink = igb_del_vf,
+};
#endif
static pci_ers_result_t igb_io_error_detected(struct pci_dev *,

		     pci_channel_state_t);

@@ -304,11 +310,18 @@ static int __init igb_init_module(void)

	global_quad_port_a = 0;

+#ifdef CONFIG_PCI_IOV
+ /* since iov functionality isn't critical to base device function we
+ * can accept failure. If it fails we can't enable sr-iov */
+ rtnl_link_register(&igbpf_link_ops);
+
+#endif

#ifdef CONFIG_IGB_DCA
	dca_register_notify(&dca_notifier);

#endif

	ret = pci_register_driver(&igb_driver);
+
	return ret;
}

@@ -326,6 +339,9 @@ static void __exit igb_exit_module(void)

	dca_unregister_notify(&dca_notifier);
#endif
	pci_unregister_driver(&igb_driver);

+#ifdef CONFIG_PCI_IOV
+ rtnl_link_unregister(&igbpf_link_ops);
+#endif
}

module_exit(igb_exit_module);
@@ -1235,6 +1251,11 @@ static int __devinit igb_probe(struct pci_dev *pdev,

	if (err)
		goto err_sw_init;

+#ifdef CONFIG_PCI_IOV
+ if ((hw->mac.type == e1000_82576) && !pci_enable_sriov(pdev, 0))
+ netdev->rtnl_link_ops = &igbpf_link_ops;
+
+#endif

	/* setup the private structure */
	err = igb_sw_init(adapter);
	if (err)
@@ -1394,19 +1415,6 @@ static int __devinit igb_probe(struct pci_dev *pdev,
	if (err)
		goto err_register;

-#ifdef CONFIG_PCI_IOV
- /* since iov functionality isn't critical to base device function we
- * can accept failure. If it fails we don't allow iov to be enabled */
- if (hw->mac.type == e1000_82576) {
- err = pci_enable_sriov(pdev, 0);
- if (!err)
- err = device_create_file(&netdev->dev,
- &dev_attr_num_vfs);
- if (err)
- dev_err(&pdev->dev, "Failed to initialize IOV\n");
- }
-
-#endif

#ifdef CONFIG_IGB_DCA
	if (dca_add_requester(&pdev->dev) == 0) {
		adapter->flags |= IGB_FLAG_DCA_ENABLED;
@@ -1554,6 +1562,8 @@ static void __devexit igb_remove(struct pci_dev *pdev)
	 * would have already happened in close and is redundant. */
	igb_release_hw_control(adapter);

+ netdev->rtnl_link_ops = NULL;
+

	unregister_netdev(netdev);

	if (!igb_check_reset_block(&adapter->hw))

@@ -5402,37 +5412,28 @@ static void igb_vmm_control(struct igb_adapter *adapter) }

#ifdef CONFIG_PCI_IOV
-static ssize_t igb_show_num_vfs(struct device *dev,
- struct device_attribute *attr, char *buf)
+static int igb_set_num_vfs(struct net_device *dev, unsigned int num_vfs) {
- struct igb_adapter *adapter = netdev_priv(to_net_dev(dev));
-
- return sprintf(buf, "%d\n", adapter->vfs_allocated_count);
-}
-
-static ssize_t igb_set_num_vfs(struct device *dev,
- struct device_attribute *attr,
- const char *buf, size_t count)
-{
- struct net_device *netdev = to_net_dev(dev);
- struct igb_adapter *adapter = netdev_priv(netdev);

+	struct igb_adapter *adapter = netdev_priv(dev);
	struct e1000_hw *hw = &adapter->hw;
	struct pci_dev *pdev = adapter->pdev;
-	unsigned int num_vfs, i;
+	unsigned int i;
	unsigned char mac_addr[ETH_ALEN];

- int err;
+ int err = 0;

- sscanf(buf, "%u", &num_vfs);
+
+ if (hw->mac.type != e1000_82576)
+ return -EINVAL;

	if (num_vfs > 7)
		num_vfs = 7;

	/* value unchanged do nothing */
	if (num_vfs == adapter->vfs_allocated_count)

- return count;
+ return -EINVAL;

- if (netdev->flags & IFF_UP)
- igb_close(netdev);
+ if (dev->flags & IFF_UP)
+ igb_close(dev);

	igb_reset_interrupt_capability(adapter);
	igb_free_queues(adapter);
@@ -5440,6 +5441,7 @@ static ssize_t igb_set_num_vfs(struct device *dev,
	adapter->rx_ring = NULL;
	adapter->vfs_allocated_count = 0;
+	rtnl_unlock();
	/* reclaim resources allocated to VFs since we are changing count */
	if (adapter->vf_data) {
		/* disable iov and allow time for transactions to clear */
@@ -5451,6 +5453,7 @@ static ssize_t igb_set_num_vfs(struct device *dev,
		wr32(E1000_IOVCTL, E1000_IOVCTL_REUSE_VFQ);
		msleep(100);
		dev_info(&pdev->dev, "IOV Disabled\n");
+		err = 0;
	}

	if (num_vfs) {
@@ -5458,10 +5461,12 @@ static ssize_t igb_set_num_vfs(struct device *dev,
		                           sizeof(struct vf_data_storage),
		                           GFP_KERNEL);
		if (!adapter->vf_data) {
+			err = -ENOMEM;
			dev_err(&pdev->dev, "Could not allocate VF private "
				"data - IOV enable failed\n");
		} else {
			err = pci_enable_sriov(pdev, num_vfs);
+			msleep(100);
			if (!err) {
				adapter->vfs_allocated_count = num_vfs;
				dev_info(&pdev->dev, "%d vfs allocated\n", num_vfs);
@@ -5475,15 +5480,50 @@ static ssize_t igb_set_num_vfs(struct device *dev,
			}
		}
	}

+ rtnl_lock();

	igb_set_interrupt_capability(adapter);
	igb_alloc_queues(adapter);
	igb_reset(adapter);

- if (netdev->flags & IFF_UP)
- igb_open(netdev);
+ if (dev->flags & IFF_UP)
+ igb_open(dev);

- return count;
+ return err;
+}
+
+static int igb_new_vf(struct net_device *dev, struct nlattr *tb[],
+ struct nlattr *data[])
+{
+ struct net_device *netdev;
+ struct igb_adapter *adapter;
+ int err;
+
+ netdev = __dev_get_by_index(dev_net(dev), nla_get_u32(tb[IFLA_LINK]));
+
+ if (!netdev)
+ return -ENODEV;
+
+ adapter = netdev_priv(netdev);
+ err = igb_set_num_vfs(netdev, adapter->vfs_allocated_count + 1);
+ if (!err)
+ free_netdev(dev);
+
+ return err;
+
+}
+
+static void igb_del_vf(struct net_device *dev)
+{
+ struct igb_adapter *adapter = netdev_priv(dev);
+
+ if (adapter->vfs_allocated_count > 0)
+ igb_set_num_vfs(dev, adapter->vfs_allocated_count - 1);
+}
+
+static void igb_setup_vf(struct net_device *dev)
+{
+ /* do nothing for now */
}
#endif /* CONFIG_PCI_IOV */
/* igb_main.c */

#15 lore raw

Re: [ RFC ] igb: first draft of igb rtnl_link_ops interface for vf creation (was Re: [net-next PATCH v3] igbvf: add new driver to support 82576 virtual functions)

Quoted 3 lines · Alexander Duyck

In the meantime I have been working on the rtnl_link_ops approach and I think I have a few things going but I wanted to get some input before I go much further.

First, is it ok for me to call rtnl_unlock prior to doing my settings changes on the sriov config space, followed by rtnl_lock afterwards in my newlink and dellink operations? I ask because I had to do this in order to prevent a deadlock when the pci-hotplug events fired for the vfs and called unregister/register_netdev.

No, both functions are called with the RTNL already held. I'm not sure I understand what kind of potential deadlock you're trying to avoid. The ->newlink and ->dellink functions are called (mainly) in response to userspace netlink messages and there should never be a need to change anything related to rtnl locking.

A deadlock can happen when you call rtnl_link_unregister() while holding the RTNL. There's an unlocked version (__rtnl_link_unregister) for this case.

If that doesn't answer your question, please provide more detail.

Quoted 1 line

Second is it acceptable for me to just free the netdev at the end of newlink and call delete on the PF interface directly? I ask because I don't have any use for the netdevs that are generated and I cannot call delete on specific VFs anyway since they are allocated/freed in LIFO order so I would always have to free the last one I allocated.

No, the newly created netdev is freed when returning an error, other netdevs should not be touched.

Quoted 19 lines

I have included a patch for review below that implements these changes against the current driver. Please feel free to comment.

(quoted diff) +17 -0
+static int igb_new_vf(struct net_device *dev, struct nlattr *tb[],
+                      struct nlattr *data[])
+{
+    struct net_device *netdev;
+    struct igb_adapter *adapter;
+    int err;
+
+    netdev = __dev_get_by_index(dev_net(dev), nla_get_u32(tb[IFLA_LINK]));
+
+    if (!netdev)
+        return -ENODEV;
+   
+    adapter = netdev_priv(netdev);
+    err = igb_set_num_vfs(netdev, adapter->vfs_allocated_count + 1);
+    if (!err)
+        free_netdev(dev);
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+    return err;
+
+}
+
+static void igb_del_vf(struct net_device *dev)
+{
+    struct igb_adapter *adapter = netdev_priv(dev);
+
+    if (adapter->vfs_allocated_count > 0)
+        igb_set_num_vfs(dev, adapter->vfs_allocated_count - 1);

Thats not really how this is supposed to work. Every device is an independant instance, so you can delete them in arbitrary order. If you need to assign them some device resources, you need to do this mapping internally.

#16 lore raw

Re: [ RFC ] igb: first draft of igb rtnl_link_ops interface for vf creation

Quoted 11 lines · Patrick McHardy
Quoted 3 lines · Alexander Duyck

In the meantime I have been working on the rtnl_link_ops approach and I think I have a few things going but I wanted to get some input before I go much further.

First, is it ok for me to call rtnl_unlock prior to doing my settings changes on the sriov config space, followed by rtnl_lock afterwards in my newlink and dellink operations? I ask because I had to do this in order to prevent a deadlock when the pci-hotplug events fired for the vfs and called unregister/register_netdev.

No, both functions are called with the RTNL already held. I'm not sure I understand what kind of potential deadlock you're trying to avoid. The ->newlink and ->dellink functions are called (mainly) in response to userspace netlink messages and there should never be a need to change anything related to rtnl locking.

Quoted 5 lines

A deadlock can happen when you call rtnl_link_unregister() while holding the RTNL. There's an unlocked version (__rtnl_link_unregister) for this case.

If that doesn't answer your question, please provide more detail.

So what I was seeing prior to changing the locking is that if I had the igbvf driver loaded and enabled a vf the operation would hang, and anything that tried to configure a network interface would hang as well.

The call to enable SR-IOV is contained within the newlink and dellink calls with this patch. When I change the number of VFs it will trigger PCI hotplug events where it will remove all the VFs and then add them back. As a result there are a number of register/unregister_netdev calls that are triggered by the igbvf_probe/remove calls in the igbvf driver.

Quoted 5 lines
Quoted 2 lines

Second is it acceptable for me to just free the netdev at the end of newlink and call delete on the PF interface directly? I ask because I don't have any use for the netdevs that are generated and I cannot call delete on specific VFs anyway since they are allocated/freed in LIFO order so I would always have to free the last one I allocated.

No, the newly created netdev is freed when returning an error, other netdevs should not be touched.

The problem is I have to alloc/free VFs in order. See the rest of my comments on this below.

Quoted 26 lines
Quoted 26 lines

I have included a patch for review below that implements these changes against the current driver. Please feel free to comment.

(quoted diff) +24 -0
+static int igb_new_vf(struct net_device *dev, struct nlattr *tb[],
+                      struct nlattr *data[])
+{
+    struct net_device *netdev;
+    struct igb_adapter *adapter;
+    int err;
+
+    netdev = __dev_get_by_index(dev_net(dev), nla_get_u32(tb[IFLA_LINK]));
+
+    if (!netdev)
+        return -ENODEV;
+   
+    adapter = netdev_priv(netdev);
+    err = igb_set_num_vfs(netdev, adapter->vfs_allocated_count + 1);
+    if (!err)
+        free_netdev(dev);
+
+    return err;
+
+}
+
+static void igb_del_vf(struct net_device *dev)
+{
+    struct igb_adapter *adapter = netdev_priv(dev);
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+
+    if (adapter->vfs_allocated_count > 0)
+        igb_set_num_vfs(dev, adapter->vfs_allocated_count - 1);

Thats not really how this is supposed to work. Every device is an independant instance, so you can delete them in arbitrary order. If you need to assign them some device resources, you need to do this mapping internally.

This is where it gets messy and where we don't really have any good tools for this. The problem is each VF is not independent. If I remove VFs it has to be in LIFO ordering. This is due to the fact that SR-IOV config space only allows you to specify a number of VFs, not the ordering of them, so they cannot be enabled/disabled individually.

Thanks,

Alex