Skip to contentSkip to conversation

pciehp_driver.c: PCIe hot plug for Intel iommu - call IOMMU API in hot remove

PATCH NAK Fenghua Yu 3 participants 5 messages Oct 2, 2009 – Oct 22, 2009 3 messages not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 72D8A09E39A89147BE9CFBB6F0C73C306073445396@rrsmsx501.amr.corp.intel.com

A message that is not in the archive (sent off-list or not yet received): 6E2431EDEB6AE64F827F8A1B0D0DA1A603C322CF@azsmsx501.amr.corp.intel.com

A message that is not in the archive (sent off-list or not yet received): 72D8A09E39A89147BE9CFBB6F0C73C30607344552A@rrsmsx501.amr.corp.intel.com

Fenghua Yu

author
#3 lore raw

PATCH 2/2 pciehp_driver.c: PCIe hot plug for Intel iommu - call IOMMU API in hot remove

IOMMU API iommu_detach_devcie() is called to hot remove a PCIe device.

A hot added device will be added into an IOMMU domain when it first does IOMMU op. So there is no need to add more code for hot add.

---

 drivers/pci/pci-driver.c |    8 ++++++++
 1 files changed, 8 insertions(+)
drivers/pci/pci-driver.c +8 -0
index e5d47be..b8f0592 100644
@@ -17,6 +17,7 @@
 #include <linux/slab.h>
 #include <linux/sched.h>
 #include <linux/cpu.h>
+#include <linux/iommu.h>
 #include "pci.h"
 
 struct pci_dynid {
@@ -390,6 +391,13 @@ static int pci_device_remove(struct device * dev)
 	 */
 
 	pci_dev_put(pci_dev);
+
+	/*
+	 * Remove this pci device from its iommu domain.
+	 */
+	if (iommu_found())
+		iommu_detach_device(NULL, dev);
+
 	return 0;
 }
 
#4 lore raw
Quoted 27 lines · Fenghua Yu

IOMMU API iommu_detach_devcie() is called to hot remove a PCIe device.

A hot added device will be added into an IOMMU domain when it first does IOMMU op. So there is no need to add more code for hot add.

---

 drivers/pci/pci-driver.c |    8 ++++++++
 1 files changed, 8 insertions(+)
drivers/pci/pci-driver.c +1 -0
index e5d47be..b8f0592 100644
@@ -17,6 +17,7 @@
 #include <linux/slab.h>
 #include <linux/sched.h>
 #include <linux/cpu.h>
+#include <linux/iommu.h>
 #include "pci.h"
 
 struct pci_dynid {
@@ -390,6 +391,13 @@ static int pci_device_remove(struct device * dev)
 	 */
 

drivers/pci/pci-driver.c static int pci_device_remove(struct device * dev) · line 402

Quoted 10 lines
drivers/pci/pci-driver.c +7 -0
 	pci_dev_put(pci_dev);
+
+	/*
+	 * Remove this pci device from its iommu domain.
+	 */
+	if (iommu_found())
+		iommu_detach_device(NULL, dev);
+
 	return 0;
 }

NACK!

This breaks with AMD IOMMU driver.
Just implement a VT-d specific notifier callback for device removal on the pci bus.

Joerg

author
#5 lore raw
Quoted 5 lines

NACK!

This breaks with AMD IOMMU driver.
Just implement a VT-d specific notifier callback for device removal on the pci bus.

You are right. Will change to AMD's method.

Thanks.

-Fenghua

author
#6 lore raw

PATCH 1/1 PCIe hot plug for Intel iommu

In reply to Joerg Roedel

To support PCIe hot plug in IOMMU, we register a notifier to respond to device change action.

When the notifier gets BUS_NOTIFY_UNBOUND_DRIVER, it removes the device from its DMAR domain.

A hot added device will be added into an IOMMU domain when it first does IOMMU op. So there is no need to add more code for hot add.

---

 intel-iommu.c |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+)
drivers/pci/intel-iommu.c +29 -0
index 855dd7c..d8b8cfc 100644
@@ -3193,6 +3193,33 @@ static int __init init_iommu_sysfs(void)
 }
 #endif	/* CONFIG_PM */
 
+/*
+ * Here we only respond to action of unbound device from driver.
+ *
+ * Added device is not attached to its DMAR domain here yet. That will happen
+ * when mapping the device to iova.
+ */
+static int device_notifier(struct notifier_block *nb,
+				  unsigned long action, void *data)
+{
+	struct device *dev = data;
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct dmar_domain *domain;
+
+	domain = find_domain(pdev);
+	if (!domain)
+		return 0;
+
+	if (action == BUS_NOTIFY_UNBOUND_DRIVER && !iommu_pass_through)
+		domain_remove_one_dev_info(domain, pdev);
+
+	return 0;
+}
+
+static struct notifier_block device_nb = {
+	.notifier_call = device_notifier,
+};
+
 int __init intel_iommu_init(void)
 {
 	int ret = 0;
@@ -3245,6 +3272,8 @@ int __init intel_iommu_init(void)
 
 	register_iommu(&intel_iommu_ops);
 
+	bus_register_notifier(&pci_bus_type, &device_nb);
+
 	return 0;
 }
 
#7 lore raw

Re: [PATCH] PCIe hot plug for Intel iommu

Quoted 55 lines · Fenghua Yu

To support PCIe hot plug in IOMMU, we register a notifier to respond to device change action.

When the notifier gets BUS_NOTIFY_UNBOUND_DRIVER, it removes the device from its DMAR domain.

A hot added device will be added into an IOMMU domain when it first does IOMMU op. So there is no need to add more code for hot add.

---

 intel-iommu.c |   29 +++++++++++++++++++++++++++++

 1 files changed, 29 insertions(+)
drivers/pci/intel-iommu.c +27 -0
index 855dd7c..d8b8cfc 100644
@@ -3193,6 +3193,33 @@ static int __init init_iommu_sysfs(void)
  }
  #endif /* CONFIG_PM */
 
+/*
+ * Here we only respond to action of unbound device from driver.
+ *
+ * Added device is not attached to its DMAR domain here yet. That will happen
+ * when mapping the device to iova.
+ */
+static int device_notifier(struct notifier_block *nb,
+                                 unsigned long action, void *data)
+{
+       struct device *dev = data;
+       struct pci_dev *pdev = to_pci_dev(dev);
+       struct dmar_domain *domain;
+
+       domain = find_domain(pdev);
+       if (!domain)
+               return 0;
+
+       if (action == BUS_NOTIFY_UNBOUND_DRIVER && !iommu_pass_through)
+               domain_remove_one_dev_info(domain, pdev);
+
+       return 0;
+}
+
+static struct notifier_block device_nb = {
+       .notifier_call = device_notifier,
+};
+
  int __init intel_iommu_init(void)
  {
        int ret = 0;

drivers/pci/intel-iommu.c int __init intel_iommu_init(void) · line 3279

Quoted 8 lines
drivers/pci/intel-iommu.c +2 -0
@@ -3245,6 +3272,8 @@ int __init intel_iommu_init(void)
 
        register_iommu(&intel_iommu_ops);
 
+       bus_register_notifier(&pci_bus_type, &device_nb);
+
        return 0;
  }
 

it seems this one should be get into 2.6.32....and .stable

otherwise second hot plug in the same pcie slot will not work with dma.

YH