IOMMU probe deferral support

Configuring DMA ops at probe time will allow deferring device probe when the IOMMU isn't available yet. The dma_configure for the device is now called from the generic device_attach callback just before the bus/driver probe is called. This way, configuring the dma ops for the device would be called at the same place for all bus_types, hence the deferred probing mechanism should work for all buses as well.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

Signed-off-by Sricharan R <[email protected]>

---
 drivers/base/dd.c           | 10 ++++++++++
 drivers/base/dma-mapping.c  | 11 +++++++++++
 drivers/of/platform.c       |  4 ----
 drivers/pci/probe.c         |  5 +----
 include/linux/dma-mapping.h |  3 +++
 5 files changed, 25 insertions(+), 8 deletions(-)

drivers/base/dd.c +10 -0

index 16688f5..cfebd48 100644
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>

@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));

@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);

@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);

drivers/base/dma-mapping.c +11 -0

index d799662..54e87f5 100644
@@ -10,6 +10,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
 }
 EXPORT_SYMBOL(dmam_free_noncoherent);
 
+int dma_configure(struct device *dev)
+{
+	return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+}
+
 #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
 
 static void dmam_coherent_decl_release(struct device *dev, void *res)

drivers/of/platform.c +0 -4

index 9cb7090..adbd77c 100644
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}

@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);

@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;

drivers/pci/probe.c +0 -0

index 93f280d..85c9553 100644

include/linux/dma-mapping.h +3 -0

index 66533e1..2766dbe 100644
@@ -656,6 +656,9 @@ dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */

Quoted 141 lines

Configuring DMA ops at probe time will allow deferring device probe when the IOMMU isn't available yet. The dma_configure for the device is now called from the generic device_attach callback just before the bus/driver probe is called. This way, configuring the dma ops for the device would be called at the same place for all bus_types, hence the deferred probing mechanism should work for all buses as well.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

Signed-off-by Sricharan R <[email protected]>

---
 drivers/base/dd.c           | 10 ++++++++++
 drivers/base/dma-mapping.c  | 11 +++++++++++
 drivers/of/platform.c       |  4 ----
 drivers/pci/probe.c         |  5 +----
 include/linux/dma-mapping.h |  3 +++
 5 files changed, 25 insertions(+), 8 deletions(-)

drivers/base/dd.c +10 -0

index 16688f5..cfebd48 100644
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>

@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));

@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);

@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);

drivers/base/dma-mapping.c +11 -0

index d799662..54e87f5 100644
@@ -10,6 +10,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
 }
 EXPORT_SYMBOL(dmam_free_noncoherent);
 
+int dma_configure(struct device *dev)
+{
+	return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+}
+
 #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
 
 static void dmam_coherent_decl_release(struct device *dev, void *res)

drivers/of/platform.c +0 -4

index 9cb7090..adbd77c 100644
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}

@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);

@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;

drivers/pci/probe.c +0 -0

index 93f280d..85c9553 100644

drivers/pci/probe.c static void pci_dma_configure(struct pci_dev *dev) · line 1729

Quoted 10 lines

drivers/pci/probe.c +1 -4

@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
 {
 	struct device *bridge = pci_get_host_bridge_device(dev);
 
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
+	if (has_acpi_companion(bridge)) {
 		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
 		enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA configuration at device add, after we've allegedly moved DMA configuration to driver probe. Lorenzo, do you foresee any issues if this probe-time of_dma_configure() path were to also multiplex acpi_dma_configure() in future, such that everything would be back in the same place eventually?

Conversely, is there actually any issue with leaving pci_dma_configure() unchanged, and simply moving the call from pci_device_add() into dma_configure()?

Hi Robin,

Quoted 160 lines · Sricharan R

Quoted 141 lines

Configuring DMA ops at probe time will allow deferring device probe when the IOMMU isn't available yet. The dma_configure for the device is now called from the generic device_attach callback just before the bus/driver probe is called. This way, configuring the dma ops for the device would be called at the same place for all bus_types, hence the deferred probing mechanism should work for all buses as well.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

Signed-off-by Sricharan R <[email protected]>

---
 drivers/base/dd.c           | 10 ++++++++++
 drivers/base/dma-mapping.c  | 11 +++++++++++
 drivers/of/platform.c       |  4 ----
 drivers/pci/probe.c         |  5 +----
 include/linux/dma-mapping.h |  3 +++
 5 files changed, 25 insertions(+), 8 deletions(-)

drivers/base/dd.c +10 -0

index 16688f5..cfebd48 100644
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>

@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));

@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);

@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);

drivers/base/dma-mapping.c +11 -0

index d799662..54e87f5 100644
@@ -10,6 +10,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
 }
 EXPORT_SYMBOL(dmam_free_noncoherent);
 
+int dma_configure(struct device *dev)
+{
+	return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+}
+
 #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
 
 static void dmam_coherent_decl_release(struct device *dev, void *res)

drivers/of/platform.c +0 -4

index 9cb7090..adbd77c 100644
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}

@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);

@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;

drivers/pci/probe.c +0 -0

index 93f280d..85c9553 100644

drivers/pci/probe.c static void pci_dma_configure(struct pci_dev *dev) · line 1729

Quoted 10 lines

drivers/pci/probe.c +1 -4

@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
 {
 	struct device *bridge = pci_get_host_bridge_device(dev);
 
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
+	if (has_acpi_companion(bridge)) {
 		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
 		enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA configuration at device add, after we've allegedly moved DMA configuration to driver probe. Lorenzo, do you foresee any issues if this probe-time of_dma_configure() path were to also multiplex acpi_dma_configure() in future, such that everything would be back in the same place eventually?

Quoted 3 lines

Conversely, is there actually any issue with leaving pci_dma_configure() unchanged, and simply moving the call from pci_device_add() into dma_configure()?

Ya i removed only the CONFIG_OF part out of this, and was hoping that the acpi configure can also be called from the dma_configure during probe later. I did not have an ACPI based platform though. As you said, if that looks right to the ACPI world, it can be moved out from here. I can try mergin series (after fixing the vfio errors part) with the ACPI IO port and see how it behaves.

-----Original Message-----

Sent: Thursday, October 27, 2016 4:19 PM
To: Sricharan <[email protected]>
Cc: 'Robin Murphy' <[email protected]>; will.deacon at arm.com; joro at 8bytes.org; iommu at lists.linux-foundation.org; linux- arm-kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; laurent.pinchart at ideasonboard.com; m.szyprowski at samsung.com; tfiga at chromium.org; srinivas.kandagatla at linaro.org
Subject: Re: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

Quoted 166 lines · Sricharan

Hi Robin,

Quoted 160 lines · Sricharan R

Quoted 141 lines

Configuring DMA ops at probe time will allow deferring device probe when the IOMMU isn't available yet. The dma_configure for the device is now called from the generic device_attach callback just before the bus/driver probe is called. This way, configuring the dma ops for the device would be called at the same place for all bus_types, hence the deferred probing mechanism should work for all buses as well.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

Signed-off-by Sricharan R <[email protected]>

---
 drivers/base/dd.c           | 10 ++++++++++
 drivers/base/dma-mapping.c  | 11 +++++++++++
 drivers/of/platform.c       |  4 ----
 drivers/pci/probe.c         |  5 +----
 include/linux/dma-mapping.h |  3 +++
 5 files changed, 25 insertions(+), 8 deletions(-)

drivers/base/dd.c +10 -0

index 16688f5..cfebd48 100644
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>

@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));

@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);

@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);

drivers/base/dma-mapping.c +11 -0

index d799662..54e87f5 100644
@@ -10,6 +10,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
 }
 EXPORT_SYMBOL(dmam_free_noncoherent);
 
+int dma_configure(struct device *dev)
+{
+	return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+}
+
 #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
 
 static void dmam_coherent_decl_release(struct device *dev, void *res)

drivers/of/platform.c +0 -4

index 9cb7090..adbd77c 100644
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}

@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);

@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;

drivers/pci/probe.c +0 -0

index 93f280d..85c9553 100644

drivers/pci/probe.c static void pci_dma_configure(struct pci_dev *dev) · line 1729

Quoted 10 lines

drivers/pci/probe.c +1 -4

@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
 {
 	struct device *bridge = pci_get_host_bridge_device(dev);
 
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
+	if (has_acpi_companion(bridge)) {
 		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
 		enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA configuration at device add, after we've allegedly moved DMA configuration to driver probe. Lorenzo, do you foresee any issues if this probe-time of_dma_configure() path were to also multiplex acpi_dma_configure() in future, such that everything would be back in the same place eventually?

Quoted 3 lines

Conversely, is there actually any issue with leaving pci_dma_configure() unchanged, and simply moving the call from pci_device_add() into dma_configure()?

Quoted 6 lines

Ya i removed only the CONFIG_OF part out of this, and was hoping that the acpi configure can also be called from the dma_configure during probe later. I did not have an ACPI based platform though. As you said, if that looks right to the ACPI world, it can be moved out from here. I can try mergin series (after fixing the vfio errors part) with the ACPI IO port and see how it behaves.

I do not expect any issue from this change, as long as, as Robin said, it is done consistently which means that I am not ok with this patch as it stands (ie you should defer pci_dma_configure() in its entirety, not just the DT bits, which obviously requires some ACPI testing too).

CC me in please in the next posting since this affects the ACPI probing path too, we have to coordinate this series with my ACPI IORT SMMU enablement patches:

https://lkml.org/lkml/2016/10/18/506

From: Laurent Pinchart <[email protected]>

Failures to look up an IOMMU when parsing the DT iommus property need to be handled separately from the .of_xlate() failures to support deferred probing.

The lack of a registered IOMMU can be caused by the lack of a driver for the IOMMU, the IOMMU device probe not having been performed yet, having been deferred, or having failed.

The first case occurs when the device tree describes the bus master and IOMMU topology correctly but no device driver exists for the IOMMU yet or the device driver has not been compiled in. Return NULL, the caller will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus master device as well. A mechanism to either configure the bus master device without an IOMMU or to fail the bus master device probe depending on whether the IOMMU is optional or mandatory would be a good enhancement.

The current iommu framework handles pci and non-pci devices separately, so taken care of both the paths in this patch. The iommu's add_device callback is invoked after the master's configuration data is added in xlate. This is needed because the iommu core calls add_device callback during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually that call has to be removed.

Signed-off-by Sricharan R <[email protected]>

---
 drivers/iommu/of_iommu.c  | 47 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/of/device.c       |  7 ++++++-
 include/linux/of_device.h |  6 ++++--
 3 files changed, 53 insertions(+), 7 deletions(-)

drivers/iommu/of_iommu.c +1 -0

index 5b82862..1a5e28b 100644
@@ -23,6 +23,7 @@
 #include <linux/of.h>
 #include <linux/of_iommu.h>
 #include <linux/of_pci.h>
+#include <linux/pci.h>
 #include <linux/slab.h>
 
 static const struct of_device_id __iommu_of_table_sentinel

drivers/iommu/of_iommu.c +3 -0

+	}
+
 	of_node_put(iommu_spec.np);
+
 	return ops;
 }
 

@@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;

drivers/iommu/of_iommu.c +1 -0

+	}
 
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.

drivers/iommu/of_iommu.c +1 -0

+
 		if (!ops || !ops->of_xlate ||
 		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
 		    ops->of_xlate(dev, &iommu_spec))

drivers/iommu/of_iommu.c +2 -2

 	return ops;
 
 err_put_node:
 	of_node_put(np);
-	return NULL;
+	return ops;
 }
 
 static int __init of_iommu_init(void)

@@ -222,7 +261,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}

drivers/of/device.c +6 -1

index 1c843e2..c8e74d7 100644
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;

@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 

include/linux/of_device.h +4 -2

index d20a31a..2bdb872 100644
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 

@@ -100,7 +100,9 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return NULL;
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */

Quoted 24 lines

From: Laurent Pinchart <[email protected]>

Failures to look up an IOMMU when parsing the DT iommus property need to be handled separately from the .of_xlate() failures to support deferred probing.

The lack of a registered IOMMU can be caused by the lack of a driver for the IOMMU, the IOMMU device probe not having been performed yet, having been deferred, or having failed.

The first case occurs when the device tree describes the bus master and IOMMU topology correctly but no device driver exists for the IOMMU yet or the device driver has not been compiled in. Return NULL, the caller will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus master device as well. A mechanism to either configure the bus master device without an IOMMU or to fail the bus master device probe depending on whether the IOMMU is optional or mandatory would be a good enhancement.

Quoted 6 lines

The current iommu framework handles pci and non-pci devices separately, so taken care of both the paths in this patch. The iommu's add_device callback is invoked after the master's configuration data is added in xlate. This is needed because the iommu core calls add_device callback during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually that call has to be removed.

Laurent's signoff seems to have gone missing here.

Quoted 23 lines

Signed-off-by Sricharan R <[email protected]>

---
 drivers/iommu/of_iommu.c  | 47 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/of/device.c       |  7 ++++++-
 include/linux/of_device.h |  6 ++++--
 3 files changed, 53 insertions(+), 7 deletions(-)

drivers/iommu/of_iommu.c +1 -0

index 5b82862..1a5e28b 100644
@@ -23,6 +23,7 @@
 #include <linux/of.h>
 #include <linux/of_iommu.h>
 #include <linux/of_pci.h>
+#include <linux/pci.h>
 #include <linux/slab.h>
 
 static const struct of_device_id __iommu_of_table_sentinel

@@ -167,12 +168,29 @@ static const struct iommu_ops
 		return NULL;
 

drivers/iommu/of_iommu.c static const struct iommu_ops · line 176

Quoted 7 lines

drivers/iommu/of_iommu.c +6 -0

 	ops = of_iommu_get_ops(iommu_spec.np);
+
+	if (!ops) {
+		const struct of_device_id *oid;
+
+		oid = of_match_node(&__iommu_of_table, iommu_spec.np);
+		ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;

It would seem even simpler and more convenient to roll this into the end of of_iommu_get_ops():

	if (!ops && of_match_node(&__iommu_of_table, iommu_spec.np))
		ops = ERR_PTR(-EPROBE_DEFER);

then just fix up the existing !ops checks to !IS_ERR(ops) appropriately.

Quoted 9 lines

drivers/iommu/of_iommu.c +3 -0

+		return ops;
+	}
+
 	if (!ops || !ops->of_xlate ||
 	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
 	    ops->of_xlate(&pdev->dev, &iommu_spec))
 		ops = NULL;
 

drivers/iommu/of_iommu.c static const struct iommu_ops · line 186

Quoted 2 lines

drivers/iommu/of_iommu.c +2 -0

+	if (ops && ops->add_device) {
+		ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;

This is a really obtuse way of writing

	if (ops->add_device(...))
		ops = NULL;

However, given that we're now returning an ERR_PTR, it would be worth capturing the return value of add_device and propagating the error back up - if the IOMMU driver has refused one of its masters for some reason, it probably isn't safe to allow that device to do DMA either way, so we ought to prevent it probing at all.

drivers/iommu/of_iommu.c static const struct iommu_ops · line 189

Quoted 3 lines

drivers/iommu/of_iommu.c +3 -0

+
+		if (!ops)
+			dev_err(&pdev->dev, "Failed to setup iommu ops\n");

Given the above, I think this should be more along the lines of "Device rejected by IOMMU: %d" with the actual error code as well. It's one of those "if you ever see it, you're going to have to debug it" cases, so the clearer the better.

Quoted 16 lines

drivers/iommu/of_iommu.c +7 -0

+	}
+
 	of_node_put(iommu_spec.np);
+
 	return ops;
 }
 

@@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
+	struct device *bridge;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
 

drivers/iommu/of_iommu.c const struct iommu_ops *of_iommu_configure(struct device *dev, · line 211

Quoted 5 lines

drivers/iommu/of_iommu.c +3 -2

-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
+		if (bridge && bridge->parent && bridge->parent->of_node)
+			return of_pci_iommu_configure(to_pci_dev(dev),
+						      bridge->parent->of_node);

else fall through to treating it as a platform device?

...that's not right. Anyway, this is PCI-specific stuff so should be in the PCI-specific helper function.

Quoted 8 lines

drivers/iommu/of_iommu.c +1 -0

+	}
 
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.

@@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 		np = iommu_spec.np;
 		ops = of_iommu_get_ops(np);
 

drivers/iommu/of_iommu.c const struct iommu_ops *of_iommu_configure(struct device *dev, · line 231

Quoted 7 lines

drivers/iommu/of_iommu.c +7 -0

+		if (!ops) {
+			const struct of_device_id *oid;
+
+			oid = of_match_node(&__iommu_of_table, np);
+			ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+			goto err_put_node;
+		}

Same comment as above. Especially since moving it to of_iommu_get_ops() would obviate the duplication.

Quoted 9 lines

drivers/iommu/of_iommu.c +1 -0

+
 		if (!ops || !ops->of_xlate ||
 		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
 		    ops->of_xlate(dev, &iommu_spec))

@@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 		idx++;
 	}
 

drivers/iommu/of_iommu.c const struct iommu_ops *of_iommu_configure(struct device *dev, · line 248

Quoted 7 lines

drivers/iommu/of_iommu.c +7 -0

+	if (ops && ops->add_device) {
+		ops = (ops->add_device(dev) == 0) ? ops : NULL;
+
+		if (!ops)
+			dev_err(dev, "Failed to setup iommu_ops\n");
+	}
+

It would be nice to avoid duplicating this as well.

The dma_ops for the device is not getting set to NULL in arch_tear_down_dma_ops and this causes an issue when the device's probe gets deferred and retried. So reset the dma_ops to NULL.

Signed-off-by Sricharan R <[email protected]>

---
 arch/arm/mm/dma-mapping.c | 1 +
 1 file changed, 1 insertion(+)

arch/arm/mm/dma-mapping.c +1 -0

index dde6514..b9191f0 100644
@@ -2295,6 +2295,7 @@ static void arm_teardown_iommu_dma_ops(struct device *dev)
 
 	__arm_iommu_detach_device(dev);
 	arm_iommu_release_mapping(mapping);
+	set_dma_ops(dev, NULL);
 }
 
 #else

-----Original Message-----
From: Robin Murphy [mailto:robin.murphy at arm.com]
Sent: Wednesday, October 26, 2016 8:37 PM
To: Sricharan R <[email protected]>; will.deacon at arm.com; joro at 8bytes.org; iommu at lists.linux-foundation.org; linux-arm- kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; laurent.pinchart at ideasonboard.com; m.szyprowski at samsung.com; tfiga at chromium.org; srinivas.kandagatla at linaro.org
Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

Quoted 4 lines · Sricharan R

The dma_ops for the device is not getting set to NULL in arch_tear_down_dma_ops and this causes an issue when the device's probe gets deferred and retried. So reset the dma_ops to NULL.

Reviewed-by Robin Murphy <[email protected]>

This seems like it could stand independently from the rest of the series
- might be worth rewording the commit message to more general terms, i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops() thus clearing the ops set by the latter, and sending it to Russell as a fix in its own right.

Hi Robin,

Quoted 8 lines

-----Original Message-----
From: Robin Murphy [mailto:robin.murphy at arm.com]
Sent: Wednesday, October 26, 2016 8:37 PM
To: Sricharan R <[email protected]>; will.deacon at arm.com; joro at 8bytes.org; iommu at lists.linux-foundation.org; linux-arm- kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; laurent.pinchart at ideasonboard.com; m.szyprowski at samsung.com; tfiga at chromium.org; srinivas.kandagatla at linaro.org
Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

Quoted 8 lines

Quoted 4 lines · Sricharan R

The dma_ops for the device is not getting set to NULL in arch_tear_down_dma_ops and this causes an issue when the device's probe gets deferred and retried. So reset the dma_ops to NULL.

Reviewed-by Robin Murphy <[email protected]>

Thanks.

Quoted 5 lines

This seems like it could stand independently from the rest of the series
- might be worth rewording the commit message to more general terms, i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops() thus clearing the ops set by the latter, and sending it to Russell as a fix in its own right.

Ok, will reword the commit log and push this separately.

Quoted 21 lines · Sricharan

Hi Robin,

Quoted 8 lines

-----Original Message-----
From: Robin Murphy [mailto:robin.murphy at arm.com]
Sent: Wednesday, October 26, 2016 8:37 PM
To: Sricharan R <[email protected]>; will.deacon at arm.com; joro at 8bytes.org; iommu at lists.linux-foundation.org; linux-arm- kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; laurent.pinchart at ideasonboard.com; m.szyprowski at samsung.com; tfiga at chromium.org; srinivas.kandagatla at linaro.org
Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

Quoted 8 lines

Quoted 4 lines · Sricharan R

The dma_ops for the device is not getting set to NULL in arch_tear_down_dma_ops and this causes an issue when the device's probe gets deferred and retried. So reset the dma_ops to NULL.

Reviewed-by Robin Murphy <[email protected]>

Thanks.

Quoted 7 lines

Quoted 5 lines

This seems like it could stand independently from the rest of the series
- might be worth rewording the commit message to more general terms, i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops() thus clearing the ops set by the latter, and sending it to Russell as a fix in its own right.

Ok, will reword the commit log and push this separately.

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops

arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

Quoted 23 lines · Russell King - ARM Linux

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops

arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

Quoted 6 lines

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since its introduction in 4bb25789ed28. However, that has clearly not proven to be an issue in practice since then. With 09515ef5ddad we start actually calling arch_teardown_dma_ops() in a manner that might leave things partially initialised if anyone starts removing and reprobing drivers, but so far that's still from code inspection[1] rather than anyone hitting it.

Given that the changes which tickle it are fresh in -rc1 I'd say there's no need to backport this, but at the same time it shouldn't do any damage if you really want to.

Quoted 39 lines · Robin Murphy

Quoted 23 lines · Russell King - ARM Linux

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops
arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

Quoted 6 lines

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since its introduction in 4bb25789ed28. However, that has clearly not proven to be an issue in practice since then. With 09515ef5ddad we start actually calling arch_teardown_dma_ops() in a manner that might leave things partially initialised if anyone starts removing and reprobing drivers, but so far that's still from code inspection[1] rather than anyone hitting it.

Quoted 3 lines

Given that the changes which tickle it are fresh in -rc1 I'd say there's no need to backport this, but at the same time it shouldn't do any damage if you really want to.

Well, looking at this, I'm not convinced that much of it is correct.

1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops() rather than arch_teardown_dma_ops().

This doesn't strike me as being particularly symmetric. arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s counterpart.

2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops check, and Xen - Xen wants to override the DMA ops if in the initial domain. It's not clear (at least to me) whether the recent patch adding the dma_ops check took account of this or not.

3) random places seem to fiddle with the dma_ops - notice that arm_iommu_detach_device() sets the dma_ops to NULL.

In fact, I think moving __arm_iommu_detach_device() into arm_iommu_detach_device(), calling arm_iommu_detach_device(), and getting rid of the explicit set_dma_ops(, NULL) in this path would be a good first step.

4) I think arch_setup_dma_ops() is over-complex.

So, in summary, this code is a mess today, and that means it's not obviously correct - which is bad. This needs sorting.

Hi Russell,

Quoted 67 lines · On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:

Quoted 39 lines · Robin Murphy

Quoted 23 lines · Russell King - ARM Linux

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops
arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

Quoted 6 lines

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since its introduction in 4bb25789ed28. However, that has clearly not proven to be an issue in practice since then. With 09515ef5ddad we start actually calling arch_teardown_dma_ops() in a manner that might leave things partially initialised if anyone starts removing and reprobing drivers, but so far that's still from code inspection[1] rather than anyone hitting it.

Quoted 3 lines

Given that the changes which tickle it are fresh in -rc1 I'd say there's no need to backport this, but at the same time it shouldn't do any damage if you really want to.

Well, looking at this, I'm not convinced that much of it is correct.

1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops() rather than arch_teardown_dma_ops().

This doesn't strike me as being particularly symmetric. arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s counterpart.

2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops check, and Xen - Xen wants to override the DMA ops if in the initial domain. It's not clear (at least to me) whether the recent patch adding the dma_ops check took account of this or not.

3) random places seem to fiddle with the dma_ops - notice that arm_iommu_detach_device() sets the dma_ops to NULL.

In fact, I think moving __arm_iommu_detach_device() into arm_iommu_detach_device(), calling arm_iommu_detach_device(), and getting rid of the explicit set_dma_ops(, NULL) in this path would be a good first step.

Quoted 4 lines

4) I think arch_setup_dma_ops() is over-complex.

So, in summary, this code is a mess today, and that means it's not obviously correct - which is bad. This needs sorting.

We've reached the same conclusion independently, but I'll refrain from commenting on whether that's a good or bad thing :-)

I don't think this patch should be applied, as it could break Xen (and other platforms/drivers that set the DMA operations manually) by resetting DMA operations at device remove() time even if they have been set independently of arch_setup_dma_ops().

Quoted 75 lines · Laurent Pinchart

Hi Russell,

Quoted 67 lines · On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:

Quoted 39 lines · Robin Murphy

Quoted 23 lines · Russell King - ARM Linux

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops
arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

Quoted 6 lines

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since its introduction in 4bb25789ed28. However, that has clearly not proven to be an issue in practice since then. With 09515ef5ddad we start actually calling arch_teardown_dma_ops() in a manner that might leave things partially initialised if anyone starts removing and reprobing drivers, but so far that's still from code inspection[1] rather than anyone hitting it.

Quoted 3 lines

Given that the changes which tickle it are fresh in -rc1 I'd say there's no need to backport this, but at the same time it shouldn't do any damage if you really want to.

Well, looking at this, I'm not convinced that much of it is correct.

1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops() rather than arch_teardown_dma_ops().

This doesn't strike me as being particularly symmetric. arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s counterpart.

2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops check, and Xen - Xen wants to override the DMA ops if in the initial domain. It's not clear (at least to me) whether the recent patch adding the dma_ops check took account of this or not.

3) random places seem to fiddle with the dma_ops - notice that arm_iommu_detach_device() sets the dma_ops to NULL.

In fact, I think moving __arm_iommu_detach_device() into arm_iommu_detach_device(), calling arm_iommu_detach_device(), and getting rid of the explicit set_dma_ops(, NULL) in this path would be a good first step.

Quoted 4 lines

4) I think arch_setup_dma_ops() is over-complex.

So, in summary, this code is a mess today, and that means it's not obviously correct - which is bad. This needs sorting.

Quoted 7 lines

We've reached the same conclusion independently, but I'll refrain from commenting on whether that's a good or bad thing :-)

I don't think this patch should be applied, as it could break Xen (and other platforms/drivers that set the DMA operations manually) by resetting DMA operations at device remove() time even if they have been set independently of arch_setup_dma_ops().

That will only occur if the dma ops have been overriden once the DMA operations have been setup via arch_setup_dma_ops. What saves it from wholesale NULLing of the DMA operations is the check for a valid dma_iommu_mapping structure in arm_teardown_iommu_dma_ops(). This only exists when arm_setup_iommu_dma_ops() has attached a mapping to the device.

Quoted 83 lines · Russell King - ARM Linux

Quoted 81 lines · Laurent Pinchart

Quoted 68 lines · On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:

Quoted 40 lines · Robin Murphy

Quoted 24 lines · Russell King - ARM Linux

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops

arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

Quoted 6 lines

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since its introduction in 4bb25789ed28. However, that has clearly not proven to be an issue in practice since then. With 09515ef5ddad we start actually calling arch_teardown_dma_ops() in a manner that might leave things partially initialised if anyone starts removing and reprobing drivers, but so far that's still from code inspection[1] rather than anyone hitting it.

Quoted 3 lines

Given that the changes which tickle it are fresh in -rc1 I'd say there's no need to backport this, but at the same time it shouldn't do any damage if you really want to.

Well, looking at this, I'm not convinced that much of it is correct.

1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops() rather than arch_teardown_dma_ops().

This doesn't strike me as being particularly symmetric. arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s counterpart.

2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops check, and Xen - Xen wants to override the DMA ops if in the initial domain. It's not clear (at least to me) whether the recent patch adding the dma_ops check took account of this or not.

3) random places seem to fiddle with the dma_ops - notice that arm_iommu_detach_device() sets the dma_ops to NULL.

In fact, I think moving __arm_iommu_detach_device() into arm_iommu_detach_device(), calling arm_iommu_detach_device(), and getting rid of the explicit set_dma_ops(, NULL) in this path would be a good first step.

Quoted 4 lines

4) I think arch_setup_dma_ops() is over-complex.

So, in summary, this code is a mess today, and that means it's not obviously correct - which is bad. This needs sorting.

We've reached the same conclusion independently, but I'll refrain from commenting on whether that's a good or bad thing :-)

I don't think this patch should be applied, as it could break Xen (and other platforms/drivers that set the DMA operations manually) by resetting DMA operations at device remove() time even if they have been set independently of arch_setup_dma_ops().

Quoted 6 lines

That will only occur if the dma ops have been overriden once the DMA operations have been setup via arch_setup_dma_ops. What saves it from wholesale NULLing of the DMA operations is the check for a valid dma_iommu_mapping structure in arm_teardown_iommu_dma_ops(). This only exists when arm_setup_iommu_dma_ops() has attached a mapping to the device.

Right, only if the dma ops are set and no dma_iommu_mapping is created for the device, then arch_teardown_iommu_dma_ops does nothing.

Firstly, this patch for resetting the dma_ops in teardown was required only when arch_setup_dma_ops has created both the mapping and dma_ops for the device. Because mappings that were created in arch_setup_dma_ops are cleared in teardown, so dma ops should also be reset. But this can be done by calling arm_iommu_detach_device() from arm_teardown_iommu_dma_ops to avoid explicitly calling set_dma_ops again, probably this what was suggested in #3 above ?

Really sorry for the mess, but below cleanups looks required otherwise,

1) set_dma_ops is called for mach-highbank, mach-mevbu during the BUS_NOTIFY_ADD_DEVICE. That should be removed and made to come from DT path and arch_setup_dma_ops. dmabounce.c also does set_dma_ops, not very sure how to handle that (swiotlb ?), are call dmabounce_register_dev during the device's probe instead to have the dma_set_ops overriding later in probe ?

2) arm_iommu_attach_device is called from ipmmu-vmsa.c, mtk_iommu_v1.c, iommu drivers, from the iommu add_device callback, called from BUS_NOTIFY_ADD_DEVICE notifier. This is a problem because, with probe-deferral, this can be overridden in arch_setup_dma_ops during device probe and cleared in teardown path. But the add_device callback notifier is not called again when the device gets reprobed again.

With probe deferral, add_device callback also gets called from of_iommu_configure during device probe, so the above drivers should be adapted to properly register the iommu_ops to have its add_device called from of_iommu_configure path. mtk_iommu_v1.c seems to be fine, but ipmmu-vmsa.c should be adapted. otherwise if the devices attached to those iommus call arm_iommu_attach_device from its probe path to override the default ops set in arch_setup_dma_ops, then all is fine. This seems to be the case with exynos_drm_iommu.c, omap3isp/isp.c.

If the above two are done, the overridding of the default dma_ops and mapping should happen after arch_setup_dma_ops is called and also overridden every time the device gets reprobed. That should help to get rid of couple of fixes that has been added.

3) As Laurent already pointed out earlier, return error codes from some of the IOMMU apis needs to standardized.

Please let me know if its right way of doing it ?

Hello,

Quoted 84 lines · On Wednesday 24 May 2017 16:01:45 Sricharan R wrote:

Quoted 83 lines · Russell King - ARM Linux

Quoted 81 lines · Laurent Pinchart

Quoted 68 lines · On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:

Quoted 40 lines · Robin Murphy

Quoted 24 lines · Russell King - ARM Linux

So, I've come to apply this patch (since it's finally landed in the patch system), and I'm not convinced that the commit message is really up to scratch.

The current commit message looks like this:

" ARM: 8674/1: dma-mapping: Reset the device's dma_ops

arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(), dma_ops should be cleared in the teardown path. Otherwise this causes problem when the probe of device is retried after being deferred. The device's iommu structures are cleared after EPROBEDEFER error, but on the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch? Looking at the history, we have "arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()" which I would have guessed is the appropriate one, but this post-dates your patch (it's very recent, v4.12-rc recent.)

So, I need more description about the problem you were seeing when you first proposed this patch.

Quoted 6 lines

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for deferred probing?

What patch is your change trying to fix? In other words, how far back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since its introduction in 4bb25789ed28. However, that has clearly not proven to be an issue in practice since then. With 09515ef5ddad we start actually calling arch_teardown_dma_ops() in a manner that might leave things partially initialised if anyone starts removing and reprobing drivers, but so far that's still from code inspection[1] rather than anyone hitting it.

Quoted 3 lines

Given that the changes which tickle it are fresh in -rc1 I'd say there's no need to backport this, but at the same time it shouldn't do any damage if you really want to.

Well, looking at this, I'm not convinced that much of it is correct.

1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops() rather than arch_teardown_dma_ops().

This doesn't strike me as being particularly symmetric. arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s counterpart.

2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops check, and Xen - Xen wants to override the DMA ops if in the initial domain. It's not clear (at least to me) whether the recent patch adding the dma_ops check took account of this or not.

3) random places seem to fiddle with the dma_ops - notice that arm_iommu_detach_device() sets the dma_ops to NULL.

In fact, I think moving __arm_iommu_detach_device() into arm_iommu_detach_device(), calling arm_iommu_detach_device(), and getting rid of the explicit set_dma_ops(, NULL) in this path would be a good first step.

Quoted 4 lines

4) I think arch_setup_dma_ops() is over-complex.

So, in summary, this code is a mess today, and that means it's not obviously correct - which is bad. This needs sorting.

We've reached the same conclusion independently, but I'll refrain from commenting on whether that's a good or bad thing :-)

I don't think this patch should be applied, as it could break Xen (and other platforms/drivers that set the DMA operations manually) by resetting DMA operations at device remove() time even if they have been set independently of arch_setup_dma_ops().

Quoted 6 lines

Quoted 6 lines

That will only occur if the dma ops have been overriden once the DMA operations have been setup via arch_setup_dma_ops. What saves it from wholesale NULLing of the DMA operations is the check for a valid dma_iommu_mapping structure in arm_teardown_iommu_dma_ops(). This only exists when arm_setup_iommu_dma_ops() has attached a mapping to the device.

Unfortunately I don't think that's always the case. The dma_iommu_mapping is also set by direct callers of arm_iommu_attach_device(), namely

- the Renesas R-Car IOMMU driver (ipmmu-vmsa)
- the Mediatek IOMMU driver (mtk-iommu-v1)
- the Exynos DRM driver
- the OMAP3 ISP driver

All these need to be fixed, but that's not v4.12-rc material. At least in the ipmmu-vmsa case, which is the one I noticed the problem with, arm_iommu_attach_device() is called before arch_setup_dma_ops(). arch_setup_dma_ops() then exits immediately when called due to the

        if (dev->dma_ops)
                return;

check at the beginning of the function. We must ensure that in that case arch_teardown_dma_ops() will not remove the mapping or set the DMA ops to NULL, and testing to_dma_iommu_mapping(dev) won't help.

Quoted 11 lines

Right, only if the dma ops are set and no dma_iommu_mapping is created for the device, then arch_teardown_iommu_dma_ops does nothing.

Firstly, this patch for resetting the dma_ops in teardown was required only when arch_setup_dma_ops has created both the mapping and dma_ops for the device. Because mappings that were created in arch_setup_dma_ops are cleared in teardown, so dma ops should also be reset. But this can be done by calling arm_iommu_detach_device() from arm_teardown_iommu_dma_ops to avoid explicitly calling set_dma_ops again, probably this what was suggested in #3 above ?

Quoted 8 lines

Really sorry for the mess, but below cleanups looks required otherwise,

1) set_dma_ops is called for mach-highbank, mach-mevbu during the BUS_NOTIFY_ADD_DEVICE. That should be removed and made to come from DT path and arch_setup_dma_ops. dmabounce.c also does set_dma_ops, not very sure how to handle that (swiotlb ?), are call dmabounce_register_dev during the device's probe instead to have the dma_set_ops overriding later in probe ?

All this needs to be addressed, but it's definitely not v4.12-rc material.

Quoted 8 lines

2) arm_iommu_attach_device is called from ipmmu-vmsa.c, mtk_iommu_v1.c, iommu drivers, from the iommu add_device callback, called from BUS_NOTIFY_ADD_DEVICE notifier. This is a problem because, with probe-deferral, this can be overridden in arch_setup_dma_ops during device probe and cleared in teardown path. But the add_device callback notifier is not called again when the device gets reprobed again.

Quoted 8 lines

With probe deferral, add_device callback also gets called from of_iommu_configure during device probe, so the above drivers should be adapted to properly register the iommu_ops to have its add_device called from of_iommu_configure path. mtk_iommu_v1.c seems to be fine, but ipmmu-vmsa.c should be adapted. otherwise if the devices attached to those iommus call arm_iommu_attach_device from its probe path to override the default ops set in arch_setup_dma_ops, then all is fine. This seems to be the case with exynos_drm_iommu.c, omap3isp/isp.c.

Same here, this needs to be addressed, but not in v4.12-rc. We need a simpler fix for v4.12-rc.

Quoted 5 lines

If the above two are done, the overridding of the default dma_ops and mapping should happen after arch_setup_dma_ops is called and also overridden every time the device gets reprobed. That should help to get rid of couple of fixes that has been added.

Quoted 4 lines

3) As Laurent already pointed out earlier, return error codes from some of the IOMMU apis needs to standardized.

Please let me know if its right way of doing it ?

Again, the patch I propose is the simplest v4.12-rc fix I can think of, short of reverting your complete IOMMU probe deferral patch series. Let's focus on the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.

Again, the patch I propose is the simplest v4.12-rc fix I can think of, short of reverting your complete IOMMU probe deferral patch series. Let's focus on the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.

Quoted 3 lines · Laurent Pinchart

Again, the patch I propose is the simplest v4.12-rc fix I can think of, short of reverting your complete IOMMU probe deferral patch series. Let's focus on the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.

Except, I don't think it fixes the problem that Sricharan is trying to fix, namely that of DMA ops that have been setup, torn down, and are trying to be re-setup again. The issue here is that results in a stale setup, because the dma_ops are left in-place from the first iommu setup, but the iommu mapping has been disposed of.

Your patch only avoids the problem of tearing down someone else's DMA ops. We need a combination of both patches together.

What needs to happen for Sricharan's problem to be resolved is:

1. move all of __arm_iommu_detach_device() into arm_iommu_detach_device().
2. replace the __arm_iommu_detach_device() call in arm_teardown_iommu_dma_ops() with arm_iommu_detach_device().

as I don't see any need to have a different order in arm_teardown_iommu_dma_ops().

Quoted 4 lines · Laurent Pinchart

Again, the patch I propose is the simplest v4.12-rc fix I can think of, short of reverting your complete IOMMU probe deferral patch series. Let's focus on the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.

Except, I don't think it fixes the problem that Sricharan is trying to fix, namely that of DMA ops that have been setup, torn down, and are trying to be re-setup again.

The issue here is that results in a stale setup, because the dma_ops are left in-place from the first iommu setup, but the iommu mapping has been disposed of.

Your patch only avoids the problem of tearing down someone else's DMA ops. We need a combination of both patches together.

What needs to happen for Sricharan's problem to be resolved is:

1. move all of __arm_iommu_detach_device() into arm_iommu_detach_device().
2. replace the __arm_iommu_detach_device() call in arm_teardown_iommu_dma_ops() with arm_iommu_detach_device().

as I don't see any need to have a different order in arm_teardown_iommu_dma_ops().

dma_deconfigure calls arch_teardown_dma_ops in the device_detach path, which is called when the device gets detached from the driver. When the device was added, iommu's add_device callback was used to add the device in to its iommu_group and setup the device to be ready to use its iommu. Similarly, call remove_device callback to remove the device from the group and reset any other device's iommu configurations.

Signed-off-by Sricharan R <[email protected]>

---
 arch/arm/mm/dma-mapping.c   | 8 ++++++++
 arch/arm64/mm/dma-mapping.c | 7 +++++++
 2 files changed, 15 insertions(+)

arch/arm/mm/dma-mapping.c +1 -0

index b9191f0..cbe22de 100644
@@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
 static void arm_teardown_iommu_dma_ops(struct device *dev)
 {
 	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+	const struct iommu_ops *ops;
 
 	if (!mapping)

arch/arm/mm/dma-mapping.c +0 -0

 	arm_iommu_release_mapping(mapping);
 	set_dma_ops(dev, NULL);
 }

arch/arm64/mm/dma-mapping.c +7 -0

index 610d8e5..faf4b92 100644
@@ -938,6 +938,13 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 void arch_teardown_dma_ops(struct device *dev)
 {
 	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	const struct iommu_ops *ops;
+
+	if (dev->iommu_fwspec) {
+		ops = dev->iommu_fwspec->ops;
+		if (ops->remove_device)
+			ops->remove_device(dev);
+	}
 
 	if (WARN_ON(domain))
 		iommu_detach_device(domain, dev);

Quoted 26 lines

dma_deconfigure calls arch_teardown_dma_ops in the device_detach path, which is called when the device gets detached from the driver. When the device was added, iommu's add_device callback was used to add the device in to its iommu_group and setup the device to be ready to use its iommu. Similarly, call remove_device callback to remove the device from the group and reset any other device's iommu configurations.

Signed-off-by Sricharan R <[email protected]>

---
 arch/arm/mm/dma-mapping.c   | 8 ++++++++
 arch/arm64/mm/dma-mapping.c | 7 +++++++
 2 files changed, 15 insertions(+)

arch/arm/mm/dma-mapping.c +1 -0

index b9191f0..cbe22de 100644
@@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
 static void arm_teardown_iommu_dma_ops(struct device *dev)
 {
 	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+	const struct iommu_ops *ops;
 
 	if (!mapping)
 		return;
 

arch/arm/mm/dma-mapping.c static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size, · line 2304

Quoted 8 lines

arch/arm/mm/dma-mapping.c +7 -0

 	__arm_iommu_detach_device(dev);
+
+	if (dev->iommu_fwspec) {
+		ops = dev->iommu_fwspec->ops;
+		if (ops->remove_device)
+			ops->remove_device(dev);
+	}
+

Yuck. It's a little unfortunate that we have to do this at all, but I see why. Still, it should be done in common code, not duplicated across arch code, not least for symmetry with where the matching add_device happened (although I think of_dma_deconfigure() would suffice, I'm not sure we really need to add an of_iommu_deconfigure() just for this).

It's also broken for IOMMU drivers which rely on the of_iommu_configure() mechanism but have not yet been converted to use iommu_fwspec (Exynos, MSM, etc.)

-----Original Message-----
From: linux-arm-kernel [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Sricharan R
Sent: Tuesday, October 04, 2016 10:34 PM
To: will.deacon at arm.com; robin.murphy at arm.com; joro at 8bytes.org; iommu at lists.linux-foundation.org; linux-arm- kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; laurent.pinchart at ideasonboard.com; m.szyprowski at samsung.com; tfiga at chromium.org; srinivas.kandagatla at linaro.org
Cc: sricharan at codeaurora.org
Subject: [PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

With arch_setup_dma_ops now being called late during device's probe after the device's iommu is probed, the notifier trick required to handle the early setup of dma_ops before the iommu group gets created is not required. So removing the notifier's here.

Signed-off-by Sricharan R <[email protected]>

---
arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------

1 file changed, 3 insertions(+), 97 deletions(-)

arch/arm64/mm/dma-mapping.c +2 -95

index faf4b92..eb593af 100644
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
 static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 			   u64 dma_base, u64 size)
 {

@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 	return true;
 }
 
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
 static int __init __iommu_dma_init(void)
 {
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
+	return iommu_dma_init();
 }
 arch_initcall(__iommu_dma_init);
 

@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 
 	if (!ops)
 		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
+

With arch_setup_dma_ops now being called late during device's probe after the device's iommu is probed, the notifier trick required to handle the early setup of dma_ops before the iommu group gets created is not required. So removing the notifier's here.

Signed-off-by Sricharan R <[email protected]>

---
 arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
 1 file changed, 3 insertions(+), 97 deletions(-)

arch/arm64/mm/dma-mapping.c +0 -10

index faf4b92..eb593af 100644
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;

arch/arm64/mm/dma-mapping.c +1 -71

 {

@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 	return true;
 }
 
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
 static int __init __iommu_dma_init(void)
 {
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
+	return iommu_dma_init();
 }
 arch_initcall(__iommu_dma_init);
 

Quoted 4 lines

With arch_setup_dma_ops now being called late during device's probe after the device's iommu is probed, the notifier trick required to handle the early setup of dma_ops before the iommu group gets created is not required. So removing the notifier's here.

Hooray!

Quoted 27 lines

Signed-off-by Sricharan R <[email protected]>

---
 arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
 1 file changed, 3 insertions(+), 97 deletions(-)

arch/arm64/mm/dma-mapping.c +0 -14

index faf4b92..eb593af 100644
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;

arch/arm64/mm/dma-mapping.c static struct dma_map_ops iommu_dma_ops = { · line 803

Quoted 6 lines

arch/arm64/mm/dma-mapping.c +0 -4

-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
 static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 			   u64 dma_base, u64 size)

This should go as well...

Quoted 85 lines

arch/arm64/mm/dma-mapping.c +1 -71

 {

@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 	return true;
 }
 
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
 static int __init __iommu_dma_init(void)
 {
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
+	return iommu_dma_init();
 }
 arch_initcall(__iommu_dma_init);
 

@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 

arch/arm64/mm/dma-mapping.c static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, · line 927

Quoted 7 lines

arch/arm64/mm/dma-mapping.c +0 -5

 	if (!ops)
 		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.

...per this commment. It has no need to be a separate function once this is the only call site (plus it has a misleadingly inaccurate name anyway).

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

        |                         |
pci_bus_add_device     (device_add/driver_register)
        |                         |
device_attach           device_initial_probe
        |                         |
__device_attach_driver    __device_attach_driver
        |
driver_probe_device
        |
really_probe
        |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Previous post of this series [3].

[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately based on review comments that both masks and ops are required only during the device probe time.

* Reworked the series based on Generic DT bindings series [2].

* Added call to iommu's remove_device in the cleanup path for arm and arm64.

* Removed the notifier trick in arm64 to handle early device registration.

* Added reset of dma_ops in cleanup path for arm based on comments.

* Fixed the pci_iommu_configure path and tested with PCI device as well.

* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.

* Fixed few other cosmetic comments.

[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code

* Added iommu add_device callback from of_iommu_configure path

[V1]
* Initial post

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.spinics.net/lists/devicetree/msg142943.html
[3] https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg13941.html
[4] https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg13940.html

Laurent Pinchart (4):
arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error

Sricharan R (4):
drivers: platform: Configure dma operations at probe time
arm: dma-mapping: Reset the device's dma_ops
arm/arm64: dma-mapping: Call iommu's remove_device callback during
device detach
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops

  arch/arm/mm/dma-mapping.c   |  18 ++++++++
  arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
  drivers/base/dd.c           |  10 +++++
  drivers/base/dma-mapping.c  |  11 +++++
  drivers/iommu/of_iommu.c    |  47 +++++++++++++++++--
  drivers/of/address.c        |  20 ++++++++-
  drivers/of/device.c         |  34 +++++++-------
  drivers/of/platform.c       |   9 ----
  drivers/pci/probe.c         |   5 +--
  include/linux/dma-mapping.h |   3 ++
  include/linux/of_device.h   |   7 ++-
  11 files changed, 138 insertions(+), 133 deletions(-)

Hi Sricharan,

Quoted 32 lines · Sricharan R

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

        |                         |
pci_bus_add_device     (device_add/driver_register)
        |                         |
device_attach           device_initial_probe
        |                         |
__device_attach_driver    __device_attach_driver
        |
driver_probe_device
        |
really_probe
        |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

Quoted 7 lines

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Thanks for continuing work on this feature! Your can add my:

Tested-by Marek Szyprowski <[email protected]>

It works fine with Exynos SYSMMU driver, although a patch is needed to fix infinite loop due to list corruption (same element is added twice if master device fails with deferred probe):

From: Marek Szyprowski <[email protected]>
Date: Mon, 10 Oct 2016 14:22:42 +0200
Subject: [PATCH] iommu/exynos: ensure that sysmmu is added only once to its
master

Since adding IOMMU deferred probing support, of_xlate() callback might be called more than once for given master device (for example it happens when masters device driver fails with EPROBE_DEFER), so ensure that SYSMMU controller is added to its master device (owner) only once.

Signed-off-by Marek Szyprowski <[email protected]>

---
 drivers/iommu/exynos-iommu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

drivers/iommu/exynos-iommu.c +5 -2

index 30808e91b775..1525a86eb829 100644
@@ -1253,7 +1253,7 @@ static int exynos_iommu_of_xlate(struct device *dev,
 {
     struct exynos_iommu_owner *owner = dev->archdata.iommu;
     struct platform_device *sysmmu = of_find_device_by_node(spec->np);
-    struct sysmmu_drvdata *data;
+    struct sysmmu_drvdata *data, *entry;
 
     if (!sysmmu)
         return -ENODEV;

@@ -1271,6 +1271,10 @@ static int exynos_iommu_of_xlate(struct device *dev,
         dev->archdata.iommu = owner;
     }
 
+    list_for_each_entry(entry, &owner->controllers, owner_node)
+        if (entry == data)
+            return 0;
+
     list_add_tail(&data->owner_node, &owner->controllers);
     return 0;
 }
--

1.9.1

Quoted 39 lines · Sricharan R

Quoted 39 lines

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

         |                         |
pci_bus_add_device     (device_add/driver_register)
         |                         |
device_attach           device_initial_probe
         |                         |
__device_attach_driver    __device_attach_driver
         |
driver_probe_device
         |
really_probe
         |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Quoted 4 lines

Thanks for continuing work on this feature! Your can add my:

Tested-by Marek Szyprowski <[email protected]>

Thanks for testing this. So the for the below fix, the remove_device callback gets called on the dma_ops cleanup path, so would it be easy to remove the data for the device there ?

Quoted 39 lines

Quoted 39 lines

Quoted 39 lines

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

         |                         |
pci_bus_add_device     (device_add/driver_register)
         |                         |
device_attach           device_initial_probe
         |                         |
__device_attach_driver    __device_attach_driver
         |
driver_probe_device
         |
really_probe
         |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Quoted 7 lines

Quoted 4 lines

Thanks for continuing work on this feature! Your can add my:

Tested-by Marek Szyprowski <[email protected]>

Thanks for testing this. So the for the below fix, the remove_device callback gets called on the dma_ops cleanup path, so would it be easy to remove the data for the device there ?

I assumed that IOMMU driver cannot be removed reliably, so all structures that it
creates are permanent. I didn't use device_add()/device_remove() callbacks, because
in current implementation device_add() is called too late (after dma-mapping glue
triggers device_attach_iommu()).

Maybe once your patchset is merged, I will move creation and management of the all
IOMMU related structures to device_add/remove callbacks.

-----Original Message-----
From: Sricharan [mailto:sricharan at codeaurora.org]
Sent: Wednesday, October 12, 2016 11:55 AM
To: 'Marek Szyprowski' <[email protected]>; 'will.deacon at arm.com' <[email protected]>; 'robin.murphy at arm.com' <[email protected]>; 'joro at 8bytes.org' <[email protected]>; 'iommu at lists.linux-foundation.org' <[email protected] foundation.org>; 'linux-arm-kernel at lists.infradead.org' <[email protected]>; 'linux-arm-msm at vger.kernel.org' <[email protected]>; 'laurent.pinchart at ideasonboard.com' <[email protected]>; 'tfiga at chromium.org' <[email protected]>; 'srinivas.kandagatla at linaro.org' <[email protected]>
Subject: RE: [PATCH V3 0/8] IOMMU probe deferral support

Hi Marek,

Quoted 48 lines

Hi Sricharan,

Quoted 32 lines · Sricharan R

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

        |                         |
pci_bus_add_device     (device_add/driver_register)
        |                         |
device_attach           device_initial_probe
        |                         |
__device_attach_driver    __device_attach_driver
        |
driver_probe_device
        |
really_probe
        |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

Quoted 7 lines

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Thanks for continuing work on this feature! Your can add my:

Tested-by Marek Szyprowski <[email protected]>

-----Original Message-----

Sent: Monday, October 10, 2016 6:07 PM
To: Sricharan R <[email protected]>; will.deacon at arm.com; robin.murphy at arm.com; joro at 8bytes.org; iommu at lists.linux- foundation.org; linux-arm-kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; laurent.pinchart at ideasonboard.com; tfiga at chromium.org; srinivas.kandagatla at linaro.org
Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support

Hi Sricharan,

Quoted 33 lines · Sricharan R

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

        |                         |
pci_bus_add_device     (device_add/driver_register)
        |                         |
device_attach           device_initial_probe
        |                         |
__device_attach_driver    __device_attach_driver
        |
driver_probe_device
        |
really_probe
        |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver
is
called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

Quoted 10 lines

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset
while
device is attached to an domain in arch_teardown_dma_ops. But with iommu_groups created from the iommu driver, the device is
always
attached to a domain/default_domain. So so the WARN has to be
removed/handled
probably.

Thanks for continuing work on this feature! Your can add my:

Tested-by Marek Szyprowski <[email protected]>

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Previous post of this series [3].

[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately based on review comments that both masks and ops are required only during the device probe time.

* Reworked the series based on Generic DT bindings series [2].

* Added call to iommu's remove_device in the cleanup path for arm and arm64.

* Removed the notifier trick in arm64 to handle early device registration.

* Added reset of dma_ops in cleanup path for arm based on comments.

* Fixed the pci_iommu_configure path and tested with PCI device as well.

* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.

* Fixed few other cosmetic comments.

[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code

* Added iommu add_device callback from of_iommu_configure path

[V1]
* Initial post

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.spinics.net/lists/devicetree/msg142943.html
[3] https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg13941.html
[4] https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg13940.html

Laurent Pinchart (4):
arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error

Sricharan R (4):
drivers: platform: Configure dma operations at probe time
arm: dma-mapping: Reset the device's dma_ops
arm/arm64: dma-mapping: Call iommu's remove_device callback during
device detach
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops

 arch/arm/mm/dma-mapping.c   |  18 ++++++++
 arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
 drivers/base/dd.c           |  10 +++++
 drivers/base/dma-mapping.c  |  11 +++++
 drivers/iommu/of_iommu.c    |  47 +++++++++++++++++--
 drivers/of/address.c        |  20 ++++++++-
 drivers/of/device.c         |  34 +++++++-------
 drivers/of/platform.c       |   9 ----
 drivers/pci/probe.c         |   5 +--
 include/linux/dma-mapping.h |   3 ++
 include/linux/of_device.h   |   7 ++-
 11 files changed, 138 insertions(+), 133 deletions(-)

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Quoted 32 lines

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

Quoted 7 lines

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

I've finally got the chance to take a proper look at this series (sorry for the delay). First up, with these patches on top of 4.9-rc2, my little script for unbinding some PCI devices and rebinding them to VFIO now goes horribly, horribly wrong.

Robin.

[   39.901592] iommu: Removing device 0000:08:00.0 from group 0
[   39.907383] ------------[ cut here ]------------
[   39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[   39.921266] Modules linked in:
[   39.924290]
[   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[   39.931967] Hardware name: ARM Juno development board (r1) (DT)
[   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[   39.960834] sp : ffffffc974d63ca0
[   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[   40.011492] x11: 0000000000000018 x10: 000000000000000d
[   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[   40.043078]
[   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[   40.049117] Call trace:
[   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[   40.057914] 3ac0:                                   ffffffc9763f20a0
0000008000000000
[   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[   40.228075] ------------[ cut here ]------------
[   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[   40.243181] Modules linked in:
[   40.246201]
[   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.255076] Hardware name: ARM Juno development board (r1) (DT)
[   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.266787] PC is at kobject_get+0x64/0x88
[   40.270840] LR is at iommu_bus_notifier+0x40/0x110
[   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[   40.282894] sp : ffffffc974d63c00
[   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[   40.328255] x13: 0000000000000004 x12: 0000000000000030
[   40.333515] x11: 0000000000000030 x10: 0101010101010101
[   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[   40.365074]
[   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[   40.371107] Call trace:
[   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[   40.379895] 3a20:                                   ffffffc9763ebc80
0000008000000000
[   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.541324] ata1.00: disabled
[   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.558871] sd 0:0:0:0: [sda] Stopping disk
[   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.586990] Unable to handle kernel paging request at virtual address
0002003e
[   40.594702] pgd = ffffffc974c80000
[   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[   40.607694]
[   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[   40.614684] Modules linked in:
[   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.625118] Hardware name: ARM Juno development board (r1) (DT)
[   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.636841] PC is at kobject_get+0x14/0x88
[   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[   40.653387] sp : ffffffc974d63c60
[   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[   40.677720] x21: 000000000000000d x20: ffffff8008dab568
[   40.682984] x19: 0000000000020002 x18: 0000000000000000
[   40.688246] x17: 0000000000000007 x16: 0000000000000001
[   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[   40.735613]
[   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[   40.973986] Call trace:
[   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[   40.982780] 3a80:                                   0000000000020002
0000008000000000
[   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[   41.148667] ---[ end trace 35c1e743d6e6c037 ]---

Segmentation fault
/ #

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Hi Robin,

Quoted 41 lines · Sricharan R

Quoted 32 lines

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy at arm.com [2]

Quoted 7 lines

This time tested this with platform and pci device for probe deferral and reprobe on arm64 based platform. There is an issue on the cleanup path for arm64 though, where there is WARN_ON if the dma_ops is reset while device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always attached to a domain/default_domain. So so the WARN has to be removed/handled probably.

Quoted 7 lines

I've finally got the chance to take a proper look at this series (sorry for the delay). First up, with these patches on top of 4.9-rc2, my little script for unbinding some PCI devices and rebinding them to VFIO now goes horribly, horribly wrong.

Robin.

Thanks for looking in to this.
I was trying to reproduce the below with a command like this in my setup.

echo 0002\:00\:00.0 > /sys/bus/pci/devices/0002\:00\:000.0/driver/unbind0.0/driver/unbind echo 0x17cb 0x0104 > /sys/bus/pci/drivers/vfio-pci/new_id

But for me the unbind and reconfiguring/adding the iommus_ops to vfio-pci did succeed, although the WARN_ON in arch_teardown_dma_ops was there, that

could be suppresed. The vfio_pci_probe was not going through because
the pci hdr_type was not PCI_HEADER_TYPE_NORMAL.
But anyways iommu unbind/rebind seemed to be going through.

If i can get what your script is doing, i can try that and see what happens.

Regards,
Sricharan

Quoted 1 line

[   39.901592] iommu: Removing device 0000:08:00.0 from group 0

Quoted 282 lines

[   39.907383] ------------[ cut here ]------------
[   39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[   39.921266] Modules linked in:
[   39.924290]
[   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[   39.931967] Hardware name: ARM Juno development board (r1) (DT)
[   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[   39.960834] sp : ffffffc974d63ca0
[   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[   40.011492] x11: 0000000000000018 x10: 000000000000000d
[   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[   40.043078]
[   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[   40.049117] Call trace:
[   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[   40.057914] 3ac0:                                   ffffffc9763f20a0
0000008000000000
[   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[   40.228075] ------------[ cut here ]------------
[   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[   40.243181] Modules linked in:
[   40.246201]
[   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.255076] Hardware name: ARM Juno development board (r1) (DT)
[   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.266787] PC is at kobject_get+0x64/0x88
[   40.270840] LR is at iommu_bus_notifier+0x40/0x110
[   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[   40.282894] sp : ffffffc974d63c00
[   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[   40.328255] x13: 0000000000000004 x12: 0000000000000030
[   40.333515] x11: 0000000000000030 x10: 0101010101010101
[   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[   40.365074]
[   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[   40.371107] Call trace:
[   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[   40.379895] 3a20:                                   ffffffc9763ebc80
0000008000000000
[   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.541324] ata1.00: disabled
[   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.558871] sd 0:0:0:0: [sda] Stopping disk
[   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.586990] Unable to handle kernel paging request at virtual address
0002003e
[   40.594702] pgd = ffffffc974c80000
[   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[   40.607694]
[   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[   40.614684] Modules linked in:
[   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.625118] Hardware name: ARM Juno development board (r1) (DT)
[   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.636841] PC is at kobject_get+0x14/0x88
[   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[   40.653387] sp : ffffffc974d63c60
[   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[   40.677720] x21: 000000000000000d x20: ffffff8008dab568
[   40.682984] x19: 0000000000020002 x18: 0000000000000000
[   40.688246] x17: 0000000000000007 x16: 0000000000000001
[   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[   40.735613]
[   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[   40.973986] Call trace:
[   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[   40.982780] 3a80:                                   0000000000020002
0000008000000000
[   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[   41.148667] ---[ end trace 35c1e743d6e6c037 ]---

Segmentation fault
/ #

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Quoted 4 lines

Quoted 4 lines

[   39.901592] iommu: Removing device 0000:08:00.0 from group 0

Yikes, on second look, that definitely shouldn't be happening. Everything below is probably the resulting fallout.

Robin.

Quoted 278 lines

Quoted 278 lines

[   39.907383] ------------[ cut here ]------------
[   39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[   39.921266] Modules linked in:
[   39.924290]
[   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[   39.931967] Hardware name: ARM Juno development board (r1) (DT)
[   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[   39.960834] sp : ffffffc974d63ca0
[   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[   40.011492] x11: 0000000000000018 x10: 000000000000000d
[   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[   40.043078]
[   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[   40.049117] Call trace:
[   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[   40.057914] 3ac0:                                   ffffffc9763f20a0
0000008000000000
[   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[   40.228075] ------------[ cut here ]------------
[   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[   40.243181] Modules linked in:
[   40.246201]
[   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.255076] Hardware name: ARM Juno development board (r1) (DT)
[   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.266787] PC is at kobject_get+0x64/0x88
[   40.270840] LR is at iommu_bus_notifier+0x40/0x110
[   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[   40.282894] sp : ffffffc974d63c00
[   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[   40.328255] x13: 0000000000000004 x12: 0000000000000030
[   40.333515] x11: 0000000000000030 x10: 0101010101010101
[   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[   40.365074]
[   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[   40.371107] Call trace:
[   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[   40.379895] 3a20:                                   ffffffc9763ebc80
0000008000000000
[   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.541324] ata1.00: disabled
[   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.558871] sd 0:0:0:0: [sda] Stopping disk
[   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.586990] Unable to handle kernel paging request at virtual address
0002003e
[   40.594702] pgd = ffffffc974c80000
[   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[   40.607694]
[   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[   40.614684] Modules linked in:
[   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.625118] Hardware name: ARM Juno development board (r1) (DT)
[   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.636841] PC is at kobject_get+0x14/0x88
[   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[   40.653387] sp : ffffffc974d63c60
[   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[   40.677720] x21: 000000000000000d x20: ffffff8008dab568
[   40.682984] x19: 0000000000020002 x18: 0000000000000000
[   40.688246] x17: 0000000000000007 x16: 0000000000000001
[   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[   40.735613]
[   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[   40.973986] Call trace:
[   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[   40.982780] 3a80:                                   0000000000020002
0000008000000000
[   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[   41.148667] ---[ end trace 35c1e743d6e6c037 ]---

Segmentation fault
/ #

Quoted 5 lines

Quoted 4 lines

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Quoted 4 lines

Quoted 1 line

Quoted 1 line

[   39.901592] iommu: Removing device 0000:08:00.0 from group 0

Yikes, on second look, that definitely shouldn't be happening. Everything below is probably the resulting fallout.

[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops

I think the above print which says "failed to setup iommu_ops" because the call ops->add_device failed in of_pci_iommu_configure is the reason for the failure, in my case i simply do not get this even with your scripts. ops->add_device succeeds in the rebind as well. So still checking what could be happening in your case.

Regards,
Sricharan

Quoted 10 lines

Quoted 2 lines

Yikes, on second look, that definitely shouldn't be happening. Everything below is probably the resulting fallout.

[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops

I think the above print which says "failed to setup iommu_ops" because the call ops->add_device failed in of_pci_iommu_configure is the reason for the failure, in my case i simply do not get this even with your scripts. ops->add_device succeeds in the rebind as well. So still checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device callback from of_pci_iommu_configure on the rebind is the one which is causing the failure. But not able to spot out from code which point is causing the failure. It would be very helpful if i can know which is the return value from the add_device callback or point inside add_device callback which fails in your setup.

[1] git://linux-arm.org/linux-rm iommu/misc

Quoted 19 lines

Quoted 10 lines

Quoted 2 lines

Yikes, on second look, that definitely shouldn't be happening. Everything below is probably the resulting fallout.

[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops

I think the above print which says "failed to setup iommu_ops" because the call ops->add_device failed in of_pci_iommu_configure is the reason for the failure, in my case i simply do not get this even with your scripts. ops->add_device succeeds in the rebind as well. So still checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device callback from of_pci_iommu_configure on the rebind is the one which is causing the failure. But not able to spot out from code which point is causing the failure. It would be very helpful if i can know which is the return value from the add_device callback or point inside add_device callback which fails in your setup.

Quoted 1 line

[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure similar to what you reported. The issue happens when multiple devices fall in to same group due to matching sids. I ended up doing a fix like below and it would be nice to verify if it is the same that we are seeing in your setup and if the fix makes a difference ?

From: Sricharan R <[email protected]>
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting

iommu_group_get_for_dev which gets called in the add_device callback, increases the reference count of the iommu_group, so we do an iommu_group_put after that. iommu_group_get_for_dev inturn calls device_group callback and in the case of arm-smmu we call generic_device_group/pci_device_group which takes care of increasing the group's reference. But when we return an already existing group(when multiple devices have same group) the reference is not incremented, resulting in issues when the remove_device callback for the devices is invoked. Fixing the same here.

Signed-off-by Sricharan R <[email protected]>

---
 drivers/iommu/arm-smmu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

drivers/iommu/arm-smmu.c +0 -0

index 71ce4b6..a1d0b3c 100644