Skip to contentSkip to conversation

xen-pciback: drop SR-IOV VFs when PF driver unloads

PATCH In review Jan Beulich 2 participants 7 messages Nov 6, 2014 – Nov 24, 2014 Xen hypervisor interface

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen-pciback: drop SR-IOV VFs when PF driver unloads

When a PF driver unloads, it may find it necessary to leave the VFs around simply because of pciback having marked them as assigned to a guest. Utilize a suitable notification to let go of the VFs, thus allowing the PF to go back into the state it was before its driver loaded (which in particular allows the driver to be loaded again with it being able to create the VFs anew, but which also allows to then pass through the PF instead of the VFs).

Don't do this however for any VFs currently in active use by a guest.

---
 drivers/xen/xen-pciback/pci_stub.c |   46 +++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
drivers/xen/xen-pciback/pci_stub.c +46 -0
@@ -1502,6 +1502,45 @@ parse_error:
 fs_initcall(pcistub_init);
 #endif
 
+#ifdef CONFIG_PCI_IOV
+static int pci_stub_notifier(struct notifier_block *nb,
+			     unsigned long action, void *data)
+{
+	struct device *dev = data;
+	const struct pci_dev *pdev = to_pci_dev(dev);
+
+	switch (action) {
+	case BUS_NOTIFY_UNBIND_DRIVER:
+		if (!pdev->is_physfn)
+			break;
+		for (;;) {
+			struct pcistub_device *psdev;
+			unsigned long flags;
+			bool found = false;
+
+			spin_lock_irqsave(&pcistub_devices_lock, flags);
+			list_for_each_entry(psdev, &pcistub_devices, dev_list)
+				if (!psdev->pdev && psdev->dev != pdev
+				    && pci_physfn(psdev->dev) == pdev) {
+					found = true;
+					break;
+				}
+			spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+			if (!found)
+				break;
+			device_release_driver(&psdev->dev->dev);
+		}
+		break;
+	}
+
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block pci_stub_nb = {
+	.notifier_call = pci_stub_notifier,
+};
+#endif
+
 static int __init xen_pcibk_init(void)
 {
 	int err;
@@ -1523,12 +1562,19 @@ static int __init xen_pcibk_init(void)
 	err = xen_pcibk_xenbus_register();
 	if (err)
 		pcistub_exit();
+#ifdef CONFIG_PCI_IOV
+	else
+		bus_register_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 
 	return err;
 }
 
 static void __exit xen_pcibk_cleanup(void)
 {
+#ifdef CONFIG_PCI_IOV
+	bus_unregister_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 	xen_pcibk_xenbus_unregister();
 	pcistub_exit();
 }
#1 lore raw
Quoted 9 lines · Jan Beulich

When a PF driver unloads, it may find it necessary to leave the VFs around simply because of pciback having marked them as assigned to a guest. Utilize a suitable notification to let go of the VFs, thus allowing the PF to go back into the state it was before its driver loaded (which in particular allows the driver to be loaded again with it being able to create the VFs anew, but which also allows to then pass through the PF instead of the VFs).

Don't do this however for any VFs currently in active use by a guest.

I rewrote it a bit to be more in the style of pciback:

Quoted 1 line

From d364ca863f5d506750171ca014f6100080b029dc Mon Sep 17 00:00:00 2001

From: Jan Beulich <[email protected]>
Date: Thu, 6 Nov 2014 15:05:51 +0000
Subject: [PATCH] xen-pciback: drop SR-IOV VFs when PF driver unloads

When a PF driver unloads, it may find it necessary to leave the VFs around simply because of pciback having marked them as assigned to a guest. Utilize a suitable notification to let go of the VFs, thus allowing the PF to go back into the state it was before its driver loaded (which in particular allows the driver to be loaded again with it being able to create the VFs anew, but which also allows to then pass through the PF instead of the VFs).

Don't do this however for any VFs currently in active use by a guest.

[v2: Removed the switch statement, moved it about]
---
 drivers/xen/xen-pciback/pci_stub.c | 43 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
drivers/xen/xen-pciback/pci_stub.c +43 -0
index 017069a..0763e01 100644
@@ -1502,6 +1502,42 @@ parse_error:
 fs_initcall(pcistub_init);
 #endif
 
+#ifdef CONFIG_PCI_IOV
+static int pci_stub_notifier(struct notifier_block *nb,
+			     unsigned long action, void *data)
+{
+	struct device *dev = data;
+	const struct pci_dev *pdev = to_pci_dev(dev);
+	struct pcistub_device *psdev;
+	unsigned long flags;
+	bool found = false;
+
+	if (action != BUS_NOTIFY_UNBIND_DRIVER)
+		goto out;
+
+	if (!pdev->is_physfn)
+		goto out;
+
+	spin_lock_irqsave(&pcistub_devices_lock, flags);
+	list_for_each_entry(psdev, &pcistub_devices, dev_list) {
+		if (!psdev->pdev && psdev->dev != pdev
+		    && pci_physfn(psdev->dev) == pdev) {
+			found = true;
+			break;
+		}
+	}
+	spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+	if (found)
+		device_release_driver(&psdev->dev->dev);
+out:
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block pci_stub_nb = {
+	.notifier_call = pci_stub_notifier,
+};
+#endif
+
 static int __init xen_pcibk_init(void)
 {
 	int err;
@@ -1523,12 +1559,19 @@ static int __init xen_pcibk_init(void)
 	err = xen_pcibk_xenbus_register();
 	if (err)
 		pcistub_exit();
+#ifdef CONFIG_PCI_IOV
+	else
+		bus_register_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 
 	return err;
 }
 
 static void __exit xen_pcibk_cleanup(void)
 {
+#ifdef CONFIG_PCI_IOV
+	bus_unregister_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 	xen_pcibk_xenbus_unregister();
 	pcistub_exit();
 }
Signature · 5 lines
1.9.3



Haven't yet tested it.
Quoted 77 lines
---
 drivers/xen/xen-pciback/pci_stub.c |   46 +++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
drivers/xen/xen-pciback/pci_stub.c +46 -0
@@ -1502,6 +1502,45 @@ parse_error:
 fs_initcall(pcistub_init);
 #endif
 
+#ifdef CONFIG_PCI_IOV
+static int pci_stub_notifier(struct notifier_block *nb,
+			     unsigned long action, void *data)
+{
+	struct device *dev = data;
+	const struct pci_dev *pdev = to_pci_dev(dev);
+
+	switch (action) {
+	case BUS_NOTIFY_UNBIND_DRIVER:
+		if (!pdev->is_physfn)
+			break;
+		for (;;) {
+			struct pcistub_device *psdev;
+			unsigned long flags;
+			bool found = false;
+
+			spin_lock_irqsave(&pcistub_devices_lock, flags);
+			list_for_each_entry(psdev, &pcistub_devices, dev_list)
+				if (!psdev->pdev && psdev->dev != pdev
+				    && pci_physfn(psdev->dev) == pdev) {
+					found = true;
+					break;
+				}
+			spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+			if (!found)
+				break;
+			device_release_driver(&psdev->dev->dev);
+		}
+		break;
+	}
+
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block pci_stub_nb = {
+	.notifier_call = pci_stub_notifier,
+};
+#endif
+
 static int __init xen_pcibk_init(void)
 {
 	int err;
@@ -1523,12 +1562,19 @@ static int __init xen_pcibk_init(void)
 	err = xen_pcibk_xenbus_register();
 	if (err)
 		pcistub_exit();
+#ifdef CONFIG_PCI_IOV
+	else
+		bus_register_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 
 	return err;
 }
 
 static void __exit xen_pcibk_cleanup(void)
 {
+#ifdef CONFIG_PCI_IOV
+	bus_unregister_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 	xen_pcibk_xenbus_unregister();
 	pcistub_exit();
 }
#2 lore raw
In reply to Jan Beulich
Quoted 9 lines · Jan Beulich

When a PF driver unloads, it may find it necessary to leave the VFs around simply because of pciback having marked them as assigned to a guest. Utilize a suitable notification to let go of the VFs, thus allowing the PF to go back into the state it was before its driver loaded (which in particular allows the driver to be loaded again with it being able to create the VFs anew, but which also allows to then pass through the PF instead of the VFs).

Don't do this however for any VFs currently in active use by a guest.

I rewrote it a bit to be more in the style of pciback:

Quoted 1 line

From d364ca863f5d506750171ca014f6100080b029dc Mon Sep 17 00:00:00 2001

From: Jan Beulich <[email protected]>
Date: Thu, 6 Nov 2014 15:05:51 +0000
Subject: [PATCH] xen-pciback: drop SR-IOV VFs when PF driver unloads

When a PF driver unloads, it may find it necessary to leave the VFs around simply because of pciback having marked them as assigned to a guest. Utilize a suitable notification to let go of the VFs, thus allowing the PF to go back into the state it was before its driver loaded (which in particular allows the driver to be loaded again with it being able to create the VFs anew, but which also allows to then pass through the PF instead of the VFs).

Don't do this however for any VFs currently in active use by a guest.

[v2: Removed the switch statement, moved it about]
---
 drivers/xen/xen-pciback/pci_stub.c | 43 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
drivers/xen/xen-pciback/pci_stub.c +43 -0
index 017069a..0763e01 100644
@@ -1502,6 +1502,42 @@ parse_error:
 fs_initcall(pcistub_init);
 #endif
 
+#ifdef CONFIG_PCI_IOV
+static int pci_stub_notifier(struct notifier_block *nb,
+			     unsigned long action, void *data)
+{
+	struct device *dev = data;
+	const struct pci_dev *pdev = to_pci_dev(dev);
+	struct pcistub_device *psdev;
+	unsigned long flags;
+	bool found = false;
+
+	if (action != BUS_NOTIFY_UNBIND_DRIVER)
+		goto out;
+
+	if (!pdev->is_physfn)
+		goto out;
+
+	spin_lock_irqsave(&pcistub_devices_lock, flags);
+	list_for_each_entry(psdev, &pcistub_devices, dev_list) {
+		if (!psdev->pdev && psdev->dev != pdev
+		    && pci_physfn(psdev->dev) == pdev) {
+			found = true;
+			break;
+		}
+	}
+	spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+	if (found)
+		device_release_driver(&psdev->dev->dev);
+out:
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block pci_stub_nb = {
+	.notifier_call = pci_stub_notifier,
+};
+#endif
+
 static int __init xen_pcibk_init(void)
 {
 	int err;
@@ -1523,12 +1559,19 @@ static int __init xen_pcibk_init(void)
 	err = xen_pcibk_xenbus_register();
 	if (err)
 		pcistub_exit();
+#ifdef CONFIG_PCI_IOV
+	else
+		bus_register_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 
 	return err;
 }
 
 static void __exit xen_pcibk_cleanup(void)
 {
+#ifdef CONFIG_PCI_IOV
+	bus_unregister_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 	xen_pcibk_xenbus_unregister();
 	pcistub_exit();
 }
Signature · 5 lines
1.9.3



Haven't yet tested it.
Quoted 77 lines
---
 drivers/xen/xen-pciback/pci_stub.c |   46 +++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
drivers/xen/xen-pciback/pci_stub.c +46 -0
@@ -1502,6 +1502,45 @@ parse_error:
 fs_initcall(pcistub_init);
 #endif
 
+#ifdef CONFIG_PCI_IOV
+static int pci_stub_notifier(struct notifier_block *nb,
+			     unsigned long action, void *data)
+{
+	struct device *dev = data;
+	const struct pci_dev *pdev = to_pci_dev(dev);
+
+	switch (action) {
+	case BUS_NOTIFY_UNBIND_DRIVER:
+		if (!pdev->is_physfn)
+			break;
+		for (;;) {
+			struct pcistub_device *psdev;
+			unsigned long flags;
+			bool found = false;
+
+			spin_lock_irqsave(&pcistub_devices_lock, flags);
+			list_for_each_entry(psdev, &pcistub_devices, dev_list)
+				if (!psdev->pdev && psdev->dev != pdev
+				    && pci_physfn(psdev->dev) == pdev) {
+					found = true;
+					break;
+				}
+			spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+			if (!found)
+				break;
+			device_release_driver(&psdev->dev->dev);
+		}
+		break;
+	}
+
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block pci_stub_nb = {
+	.notifier_call = pci_stub_notifier,
+};
+#endif
+
 static int __init xen_pcibk_init(void)
 {
 	int err;
@@ -1523,12 +1562,19 @@ static int __init xen_pcibk_init(void)
 	err = xen_pcibk_xenbus_register();
 	if (err)
 		pcistub_exit();
+#ifdef CONFIG_PCI_IOV
+	else
+		bus_register_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 
 	return err;
 }
 
 static void __exit xen_pcibk_cleanup(void)
 {
+#ifdef CONFIG_PCI_IOV
+	bus_unregister_notifier(&pci_bus_type, &pci_stub_nb);
+#endif
 	xen_pcibk_xenbus_unregister();
 	pcistub_exit();
 }
#3 lore raw
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 21.11.14 at 23:03, <[email protected]> wrote:

I rewrote it a bit to be more in the style of pciback:
[...]
[v2: Removed the switch statement, moved it about]

What you don't mention here is that you also removed the outer loop, yet that breaks functionality afaict: There can (and I suppose normally will) be multiple VFs needing device_release_driver() called when a single PF goes away.

Also I'm not really happy for a patch with my S-o-b underneath to introduce goto-s without real need.

Jan

#4 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 21.11.14 at 23:03, <[email protected]> wrote:

I rewrote it a bit to be more in the style of pciback:
[...]
[v2: Removed the switch statement, moved it about]

What you don't mention here is that you also removed the outer loop, yet that breaks functionality afaict: There can (and I suppose normally will) be multiple VFs needing device_release_driver() called when a single PF goes away.

Also I'm not really happy for a patch with my S-o-b underneath to introduce goto-s without real need.

Jan

#5 lore raw
Quoted 9 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 21.11.14 at 23:03, <[email protected]> wrote:

I rewrote it a bit to be more in the style of pciback:
[...]
[v2: Removed the switch statement, moved it about]

What you don't mention here is that you also removed the outer loop, yet that breaks functionality afaict: There can (and I suppose normally will) be multiple VFs needing device_release_driver() called when a single PF goes away.

Good point.

Quoted 3 lines

Also I'm not really happy for a patch with my S-o-b underneath to introduce goto-s without real need.

Will ditch them.

Quoted 3 lines

Jan

#6 lore raw
In reply to Jan Beulich
Quoted 9 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 21.11.14 at 23:03, <[email protected]> wrote:

I rewrote it a bit to be more in the style of pciback:
[...]
[v2: Removed the switch statement, moved it about]

What you don't mention here is that you also removed the outer loop, yet that breaks functionality afaict: There can (and I suppose normally will) be multiple VFs needing device_release_driver() called when a single PF goes away.

Good point.

Quoted 3 lines

Also I'm not really happy for a patch with my S-o-b underneath to introduce goto-s without real need.

Will ditch them.

Quoted 3 lines

Jan