Skip to contentSkip to conversation

iommu: Retire bus_set_iommu()

PATCH Superseded Robin Murphy 7 participants 34 messages Apr 14, 2022 – Apr 23, 2022

Subsystems (18)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Robin Murphy

author reviewer
#0 lore raw

cover iommu: Retire bus_set_iommu()

Hi all,

Here's another chapter in my saga of moving to per-instance IOMMU ops - iommu_present() and iommu_capable() cleanups will be ongoing for another cycle or two, while this one is at least self-contained within the subsystem. The next steps after this are making iommu_domain_alloc() instance-aware - which should finish the public API - and pulling the fwnode/of_xlate bits into __iommu_probe_device(). And then making sense of whatever's left :)

For ease of review here I split out individual driver patches based on whether there was any non-trivial change or affect on control flow; the straightforward deletions are all lumped together since the whole series needs applying together either way, but I'm happy to split the final patch up further if anyone would like.

Patch #3 for AMD is based on Mario's SWIOTLB patch here:

https://lore.kernel.org/linux-iommu/[email protected]/

since that wants merging first as fix material. The series is also based contextually (but not functionally) on my device_iommu_capable() patches here:

https://lore.kernel.org/linux-iommu/[email protected]/

since those are pretty much good to go now (I'll send a slightly-tweaked final version once the iommu/core branch is open).

Thanks,
Robin.

Robin Murphy (13):
iommu: Always register bus notifiers
iommu: Move bus setup to IOMMU device registration iommu/amd: Clean up bus_set_iommu() iommu/arm-smmu: Clean up bus_set_iommu() iommu/arm-smmu-v3: Clean up bus_set_iommu() iommu/dart: Clean up bus_set_iommu() iommu/exynos: Clean up bus_set_iommu() iommu/ipmmu-vmsa: Clean up bus_set_iommu() iommu/mtk: Clean up bus_set_iommu() iommu/omap: Clean up bus_set_iommu() iommu/tegra-smmu: Clean up bus_set_iommu() iommu/virtio: Clean up bus_set_iommu()
iommu: Clean up bus_set_iommu()

 drivers/iommu/amd/amd_iommu.h               |   1 -
 drivers/iommu/amd/init.c                    |   9 +-
 drivers/iommu/amd/iommu.c                   |  21 ----
 drivers/iommu/apple-dart.c                  |  30 +-----
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  53 +---------
 drivers/iommu/arm/arm-smmu/arm-smmu.c       |  84 +--------------
 drivers/iommu/arm/arm-smmu/qcom_iommu.c     |   4 -
 drivers/iommu/exynos-iommu.c                |   9 --
 drivers/iommu/fsl_pamu_domain.c             |   4 -
 drivers/iommu/intel/iommu.c                 |   1 -
 drivers/iommu/iommu.c                       | 109 +++++++++-----------
 drivers/iommu/ipmmu-vmsa.c                  |  35 +------
 drivers/iommu/msm_iommu.c                   |   2 -
 drivers/iommu/mtk_iommu.c                   |  13 +--
 drivers/iommu/mtk_iommu_v1.c                |  13 +--
 drivers/iommu/omap-iommu.c                  |   6 --
 drivers/iommu/rockchip-iommu.c              |   2 -
 drivers/iommu/s390-iommu.c                  |   6 --
 drivers/iommu/sprd-iommu.c                  |   5 -
 drivers/iommu/sun50i-iommu.c                |   2 -
 drivers/iommu/tegra-smmu.c                  |  29 ++----
 drivers/iommu/virtio-iommu.c                |  24 -----
 include/linux/iommu.h                       |   1 -
 23 files changed, 62 insertions(+), 401 deletions(-)
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#1 lore raw

PATCH 1/13 iommu: Always register bus notifiers

The number of bus types that the IOMMU subsystem deals with is small and manageable, so pull that list into core code as a first step towards cleaning up all the boilerplate bus-awareness from drivers. Calling iommu_probe_device() before bus->iommu_ops is set will simply return -ENODEV and not break the notifier call chain, so there should be no harm in proactively registering all our bus notifiers at init time.

---
 drivers/iommu/iommu.c | 49 ++++++++++++++++++++++++-------------------
 1 file changed, 28 insertions(+), 21 deletions(-)
drivers/iommu/iommu.c +28 -21
index 3d6d53917e5d..13e1a8bd5435 100644
@@ -6,6 +6,7 @@
 
 #define pr_fmt(fmt)    "iommu: " fmt
 
+#include <linux/amba/bus.h>
 #include <linux/device.h>
 #include <linux/dma-iommu.h>
 #include <linux/kernel.h>
@@ -22,6 +23,7 @@
 #include <linux/err.h>
 #include <linux/pci.h>
 #include <linux/bitops.h>
+#include <linux/platform_device.h>
 #include <linux/property.h>
 #include <linux/fsl/mc.h>
 #include <linux/module.h>
@@ -74,6 +76,7 @@ static const char * const iommu_group_resv_type_string[] = {
 #define IOMMU_CMD_LINE_DMA_API		BIT(0)
 #define IOMMU_CMD_LINE_STRICT		BIT(1)
 
+static int iommu_bus_init(struct bus_type *bus);
 static int iommu_alloc_default_domain(struct iommu_group *group,
 				      struct device *dev);
 static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
@@ -102,6 +105,19 @@ struct iommu_group_attribute iommu_group_attr_##_name =		\
 static LIST_HEAD(iommu_device_list);
 static DEFINE_SPINLOCK(iommu_device_lock);
 
+static struct bus_type * const iommu_buses[] = {
+	&platform_bus_type,
+#ifdef CONFIG_PCI
+	&pci_bus_type,
+#endif
+#ifdef CONFIG_ARM_AMBA
+	&amba_bustype,
+#endif
+#ifdef CONFIG_FSL_MC_BUS
+	&fsl_mc_bus_type,
+#endif
+};
+
 /*
  * Use a function instead of an array here because the domain-type is a
  * bit-field, so an array would waste memory.
@@ -151,6 +167,10 @@ static int __init iommu_subsys_init(void)
 			(iommu_cmd_line & IOMMU_CMD_LINE_STRICT) ?
 				"(set via kernel command line)" : "");
 
+	/* If the system is so broken that this fails, it will WARN anyway */
+	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++)
+		iommu_bus_init(iommu_buses[i]);
+
 	return 0;
 }
 subsys_initcall(iommu_subsys_init);
@@ -1831,35 +1851,19 @@ int bus_iommu_probe(struct bus_type *bus)
 	return ret;
 }
 
-static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
+static int iommu_bus_init(struct bus_type *bus)
 {
 	struct notifier_block *nb;
 	int err;
 
-	nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
+	nb = kzalloc(sizeof(*nb), GFP_KERNEL);
 	if (!nb)
 		return -ENOMEM;
 
 	nb->notifier_call = iommu_bus_notifier;
-
 	err = bus_register_notifier(bus, nb);
 	if (err)
-		goto out_free;
-
-	err = bus_iommu_probe(bus);
-	if (err)
-		goto out_err;
-
-
-	return 0;
-
-out_err:
-	/* Clean up */
-	bus_for_each_dev(bus, NULL, NULL, remove_iommu_group);
-	bus_unregister_notifier(bus, nb);
-
-out_free:
-	kfree(nb);
+		kfree(nb);
 
 	return err;
 }
@@ -1892,9 +1896,12 @@ int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
 	bus->iommu_ops = ops;
 
 	/* Do IOMMU specific setup for this bus-type */
-	err = iommu_bus_init(bus, ops);
-	if (err)
+	err = bus_iommu_probe(bus);
+	if (err) {
+		/* Clean up */
+		bus_for_each_dev(bus, NULL, NULL, remove_iommu_group);
 		bus->iommu_ops = NULL;
+	}
 
 	return err;
 }
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#2 lore raw

PATCH 2/13 iommu: Move bus setup to IOMMU device registration

In reply to Robin Murphy

Move the bus setup to iommu_device_register(). This should allow bus_iommu_probe() to be correctly replayed for multiple IOMMU instances, and leaves bus_set_iommu() as a glorfied no-op to be cleaned up next.

Note that although the handling of errors from bus_iommu_probe() looks inadequate, it is merely preserving the well-established existing behaviour. This could be improved in future - probably combined with equivalent cleanup for iommu_device_unregister() - but that isn't a priority right now.

---
 drivers/iommu/iommu.c | 50 ++++++++++++++++++++++---------------------
 1 file changed, 26 insertions(+), 24 deletions(-)
drivers/iommu/iommu.c +26 -24
index 13e1a8bd5435..51205c33c426 100644
@@ -175,6 +175,14 @@ static int __init iommu_subsys_init(void)
 }
 subsys_initcall(iommu_subsys_init);
 
+static int remove_iommu_group(struct device *dev, void *data)
+{
+	if (dev->iommu && dev->iommu->iommu_dev == data)
+		iommu_release_device(dev);
+
+	return 0;
+}
+
 /**
  * iommu_device_register() - Register an IOMMU hardware instance
  * @iommu: IOMMU handle for the instance
@@ -197,6 +205,22 @@ int iommu_device_register(struct iommu_device *iommu,
 	spin_lock(&iommu_device_lock);
 	list_add_tail(&iommu->list, &iommu_device_list);
 	spin_unlock(&iommu_device_lock);
+
+	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
+		struct bus_type *bus = iommu_buses[i];
+		const struct iommu_ops *bus_ops = bus->iommu_ops;
+		int err;
+
+		WARN_ON(bus_ops && bus_ops != ops);
+		bus->iommu_ops = ops;
+		err = bus_iommu_probe(bus);
+		if (err) {
+			bus_for_each_dev(bus, NULL, iommu, remove_iommu_group);
+			bus->iommu_ops = bus_ops;
+			return err;
+		}
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(iommu_device_register);
@@ -1654,13 +1678,6 @@ static int probe_iommu_group(struct device *dev, void *data)
 	return ret;
 }
 
-static int remove_iommu_group(struct device *dev, void *data)
-{
-	iommu_release_device(dev);
-
-	return 0;
-}
-
 static int iommu_bus_notifier(struct notifier_block *nb,
 			      unsigned long action, void *data)
 {
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type *bus)
  */
 int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
 {
-	int err;
-
-	if (ops == NULL) {
-		bus->iommu_ops = NULL;
-		return 0;
-	}
-
-	if (bus->iommu_ops != NULL)
+	if (bus->iommu_ops && ops && bus->iommu_ops != ops)
 		return -EBUSY;
 
 	bus->iommu_ops = ops;
 
-	/* Do IOMMU specific setup for this bus-type */
-	err = bus_iommu_probe(bus);
-	if (err) {
-		/* Clean up */
-		bus_for_each_dev(bus, NULL, NULL, remove_iommu_group);
-		bus->iommu_ops = NULL;
-	}
-
-	return err;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(bus_set_iommu);
 
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
maintainer
#3 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 6 lines · Robin Murphy
drivers/iommu/iommu.c +0 -2
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type *bus)
   */
  int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
  {
-	int err;
-

drivers/iommu/iommu.c static int iommu_bus_init(struct bus_type *bus) · line 1906

Quoted 10 lines
drivers/iommu/iommu.c +1 -6
-	if (ops == NULL) {
-		bus->iommu_ops = NULL;
-		return 0;
-	}
-
-	if (bus->iommu_ops != NULL)
+	if (bus->iommu_ops && ops && bus->iommu_ops != ops)
  		return -EBUSY;
 
  	bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Best regards,
baolu

author reviewer
#4 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 10 lines · Lu Baolu
Quoted 9 lines · Robin Murphy
drivers/iommu/iommu.c +0 -5
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
Quoted 8 lines

drivers/iommu/iommu.c static int iommu_bus_init(struct bus_type *bus) · line 1906

Quoted 6 lines
drivers/iommu/iommu.c +1 -3
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Thanks,
Robin.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#5 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 20 lines · Robin Murphy
Quoted 10 lines · Lu Baolu
Quoted 9 lines · Robin Murphy
drivers/iommu/iommu.c +0 -5
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
Quoted 8 lines

drivers/iommu/iommu.c static int iommu_bus_init(struct bus_type *bus) · line 1906

Quoted 6 lines
drivers/iommu/iommu.c +1 -3
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Best regards,
baolu

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#6 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 21 lines · Lu Baolu
Quoted 20 lines · Robin Murphy
Quoted 18 lines · Lu Baolu
Quoted 5 lines · Robin Murphy
drivers/iommu/iommu.c +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

drivers/iommu/iommu.c static int iommu_bus_init(struct bus_type *bus) · line 1905

Quoted 10 lines
drivers/iommu/iommu.c +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

Cheers,
Robin.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#7 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 31 lines · Robin Murphy
Quoted 22 lines · Lu Baolu
Quoted 21 lines · Robin Murphy
Quoted 19 lines · Lu Baolu
Quoted 6 lines · Robin Murphy
drivers/iommu/iommu.c +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type 
 *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

drivers/iommu/iommu.c static int iommu_bus_init(struct bus_type · line 1906

Quoted 10 lines
drivers/iommu/iommu.c +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Quoted 5 lines

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

Okay, thanks for sharing.

Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#8 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

In reply to Robin Murphy

Hi Robin,

Quoted 31 lines · Robin Murphy
Quoted 22 lines · Lu Baolu
Quoted 21 lines · Robin Murphy
Quoted 19 lines · Lu Baolu
Quoted 6 lines · Robin Murphy
drivers/iommu/iommu.c +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type 
 *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

drivers/iommu/iommu.c static int iommu_bus_init(struct bus_type · line 1906

Quoted 10 lines
drivers/iommu/iommu.c +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Quoted 5 lines

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

I wound the head back to "iommu: Clean up bus_set_iommu" and tested it on an Intel machine. It got stuck during boot. This test was on a remote machine and I have no means to access it physically. So I can't get any kernel debugging messages. (I have to work from home these days. :-()

I guess it's due to the fact that intel_iommu_probe_device() callback only works for the pci devices. The issue occurs when probing a device other than a PCI one.

Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#9 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 40 lines · Lu Baolu

Hi Robin,

Quoted 31 lines · Robin Murphy
Quoted 22 lines · Lu Baolu
Quoted 21 lines · Robin Murphy
Quoted 19 lines · Lu Baolu
Quoted 6 lines · Robin Murphy
(quoted diff) +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type 
 *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

Review static int iommu_bus_init(struct bus_type · line 1906

Quoted 10 lines
(quoted diff) +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Quoted 5 lines

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

Quoted 8 lines

I wound the head back to "iommu: Clean up bus_set_iommu" and tested it on an Intel machine. It got stuck during boot. This test was on a remote machine and I have no means to access it physically. So I can't get any kernel debugging messages. (I have to work from home these days. :-()

I guess it's due to the fact that intel_iommu_probe_device() callback only works for the pci devices. The issue occurs when probing a device other than a PCI one.

Yeah, I was wondering if that would be significant, since it's the only driver that never registered itself for platform_bus_type so won't have actually seen those calls before. I'm inclined to bodge that as below for now, as long as it then works OK in terms of the rest of the changes.

Thanks,
Robin.

----->8-----

drivers/iommu/intel/iommu.c +4 -0
index 9fa1b98186a3..6e359f92ec00 100644
@@ -4565,6 +4565,10 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
 	unsigned long flags;
 	u8 bus, devfn;
 
+	/* ANDD platform device support needs fixing */
+	if (!pdev)
+		return ERR_PTR(-ENODEV);
+
 	iommu = device_to_iommu(dev, &bus, &devfn);
 	if (!iommu)
 		return ERR_PTR(-ENODEV);

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#10 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 71 lines · Robin Murphy
Quoted 40 lines · Lu Baolu

Hi Robin,

Quoted 31 lines · Robin Murphy
Quoted 22 lines · Lu Baolu
Quoted 21 lines · Robin Murphy
Quoted 19 lines · Lu Baolu
Quoted 6 lines · Robin Murphy
(quoted diff) +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct bus_type 
 *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

Review static int iommu_bus_init(struct bus_type · line 1906

Quoted 10 lines
(quoted diff) +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Quoted 5 lines

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

Quoted 8 lines

I wound the head back to "iommu: Clean up bus_set_iommu" and tested it on an Intel machine. It got stuck during boot. This test was on a remote machine and I have no means to access it physically. So I can't get any kernel debugging messages. (I have to work from home these days. :-()

I guess it's due to the fact that intel_iommu_probe_device() callback only works for the pci devices. The issue occurs when probing a device other than a PCI one.

Yeah, I was wondering if that would be significant, since it's the only driver that never registered itself for platform_bus_type so won't have actually seen those calls before. I'm inclined to bodge that as below for now, as long as it then works OK in terms of the rest of the changes.

Thanks,
Robin.

----->8-----

drivers/iommu/intel/iommu.c +4 -0
index 9fa1b98186a3..6e359f92ec00 100644
@@ -4565,6 +4565,10 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
      unsigned long flags;
      u8 bus, devfn;
 
+    /* ANDD platform device support needs fixing */
+    if (!pdev)
+        return ERR_PTR(-ENODEV);
+
      iommu = device_to_iommu(dev, &bus, &devfn);
      if (!iommu)
Quoted 1 line

         return ERR_PTR(-ENODEV);

I haven't seen any real ANDD platform devices, hence this works for me.

Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#11 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 66 lines · Lu Baolu
Quoted 65 lines · Robin Murphy
Quoted 40 lines · Lu Baolu

Hi Robin,

Quoted 31 lines · Robin Murphy
Quoted 22 lines · Lu Baolu
Quoted 21 lines · Robin Murphy
Quoted 19 lines · Lu Baolu
Quoted 6 lines · Robin Murphy
(quoted diff) +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct 
 bus_type *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

Review static int iommu_bus_init(struct · line 1906

Quoted 10 lines
(quoted diff) +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Quoted 5 lines

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

Quoted 8 lines

I wound the head back to "iommu: Clean up bus_set_iommu" and tested it on an Intel machine. It got stuck during boot. This test was on a remote machine and I have no means to access it physically. So I can't get any kernel debugging messages. (I have to work from home these days. :-()

I guess it's due to the fact that intel_iommu_probe_device() callback only works for the pci devices. The issue occurs when probing a device other than a PCI one.

Yeah, I was wondering if that would be significant, since it's the only driver that never registered itself for platform_bus_type so won't have actually seen those calls before. I'm inclined to bodge that as below for now, as long as it then works OK in terms of the rest of the changes.

Thanks,
Robin.

----->8-----

drivers/iommu/intel/iommu.c +0 -0
index 9fa1b98186a3..6e359f92ec00 100644
@@ -4565,6 +4565,10 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
       unsigned long flags;
       u8 bus, devfn;
 
Quoted 9 lines
Quoted 7 lines
drivers/iommu/intel/iommu.c +4 -0
+    /* ANDD platform device support needs fixing */
+    if (!pdev)
+        return ERR_PTR(-ENODEV);
+
       iommu = device_to_iommu(dev, &bus, &devfn);
       if (!iommu)

          return ERR_PTR(-ENODEV);

I haven't seen any real ANDD platform devices, hence this works for me.

Or more precisely, return NULL when @dev goes through device_to_iommu()?

drivers/iommu/intel/iommu.c +4 -1
index df5c62ecf942..0d447739e076 100644
@@ -797,8 +797,11 @@ struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                pf_pdev = pci_physfn(pdev);
                dev = &pf_pdev->dev;
                segment = pci_domain_nr(pdev->bus);
-       } else if (has_acpi_companion(dev))
+       } else if (has_acpi_companion(dev)) {
                dev = &ACPI_COMPANION(dev)->dev;
+       } else {
+               return NULL;
+       }
 
        rcu_read_lock();
        for_each_iommu(iommu, drhd) {

Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#12 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 96 lines · Lu Baolu
Quoted 66 lines · Lu Baolu
Quoted 65 lines · Robin Murphy
Quoted 40 lines · Lu Baolu

Hi Robin,

Quoted 31 lines · Robin Murphy
Quoted 22 lines · Lu Baolu
Quoted 21 lines · Robin Murphy
Quoted 19 lines · Lu Baolu
Quoted 6 lines · Robin Murphy
(quoted diff) +0 -1
@@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct 
 bus_type *bus)
    */
   int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
   {
-    int err;

Review static int iommu_bus_init(struct · line 1906

Quoted 10 lines
(quoted diff) +1 -7
-
-    if (ops == NULL) {
-        bus->iommu_ops = NULL;
-        return 0;
-    }
-
-    if (bus->iommu_ops != NULL)
+    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
           return -EBUSY;
       bus->iommu_ops = ops;

Do we still need to keep above lines in bus_set_iommu()?

Quoted 7 lines
Quoted 1 line

It preserves the existing behaviour until each callsite and its associated error handling are removed later on, which seems like as good a thing to do as any. Since I'm already relaxing iommu_device_register() to a warn-but-continue behaviour while it keeps the bus ops on life-support internally, I figured not changing too much at once would make it easier to bisect any potential issues arising from this first step.

Fair enough. Thank you for the explanation.

Do you have a public tree that I could pull these patches and try them on an Intel hardware? Or perhaps you have done this? I like the whole idea of this series, but it's better to try it with a real hardware.

Quoted 5 lines

I haven't bothered with separate branches since there's so many different pieces in-flight, but my complete (unstable) development branch can be found here:

https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

For now I'd recommend winding the head back to "iommu: Clean up bus_set_iommu()" for testing - some of the patches above that have already been posted and picked up by their respective subsystems, but others are incomplete and barely compile-tested. I'll probably rearrange it later this week to better reflect what's happened so far.

Quoted 8 lines

I wound the head back to "iommu: Clean up bus_set_iommu" and tested it on an Intel machine. It got stuck during boot. This test was on a remote machine and I have no means to access it physically. So I can't get any kernel debugging messages. (I have to work from home these days. :-()

I guess it's due to the fact that intel_iommu_probe_device() callback only works for the pci devices. The issue occurs when probing a device other than a PCI one.

Yeah, I was wondering if that would be significant, since it's the only driver that never registered itself for platform_bus_type so won't have actually seen those calls before. I'm inclined to bodge that as below for now, as long as it then works OK in terms of the rest of the changes.

Thanks,
Robin.

----->8-----

drivers/iommu/intel/iommu.c +0 -0
index 9fa1b98186a3..6e359f92ec00 100644
@@ -4565,6 +4565,10 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
       unsigned long flags;
       u8 bus, devfn;
 
Quoted 9 lines
Quoted 7 lines
drivers/iommu/intel/iommu.c +4 -0
+    /* ANDD platform device support needs fixing */
+    if (!pdev)
+        return ERR_PTR(-ENODEV);
+
       iommu = device_to_iommu(dev, &bus, &devfn);
       if (!iommu)

          return ERR_PTR(-ENODEV);

I haven't seen any real ANDD platform devices, hence this works for me.

Or more precisely, return NULL when @dev goes through device_to_iommu()?

drivers/iommu/intel/iommu.c +4 -1
index df5c62ecf942..0d447739e076 100644
@@ -797,8 +797,11 @@ struct intel_iommu *device_to_iommu(struct device 
 *dev, u8 *bus, u8 *devfn)
                 pf_pdev = pci_physfn(pdev);
                 dev = &pf_pdev->dev;
                 segment = pci_domain_nr(pdev->bus);
-       } else if (has_acpi_companion(dev))
+       } else if (has_acpi_companion(dev)) {
                 dev = &ACPI_COMPANION(dev)->dev;
+       } else {
+               return NULL;
+       }
 
         rcu_read_lock();
Quoted 1 line

        for_each_iommu(iommu, drhd) {

Robin, please ignore this. "has_acpi_companion(dev)" isn't equal to an ANDD device. Please use yours. Sorry for the noise.

Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

reviewer Reviewed-by on v1 2/13
#13 lore raw

RE: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

In reply to Robin Murphy

Good effort to isolate bus config from smmu drivers.

I have an orthogonal question here.
Can the following code handle the case, where different buses have different type of SMMU instances(like one bus has SMMUv2 and another bus has SMMUv3)? If it need to handle the above case, can the smmu device bus be matched with specific bus here and ops set only for that bus?

Quoted 11 lines
drivers/iommu/iommu.c +10 -0
+       for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
+               struct bus_type *bus = iommu_buses[i];
+               const struct iommu_ops *bus_ops = bus->iommu_ops;
+               int err;
+
+               WARN_ON(bus_ops && bus_ops != ops);
+               bus->iommu_ops = ops;
+               err = bus_iommu_probe(bus);
+               if (err) {
+                       bus_for_each_dev(bus, NULL, iommu,

remove_iommu_group);

drivers/iommu/iommu.c int iommu_device_register(struct iommu_device *iommu, · line 222

Quoted 4 lines
drivers/iommu/iommu.c +4 -0
+                       bus->iommu_ops = bus_ops;
+                       return err;
+               }
+       }

-KR
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#14 lore raw

Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration

Quoted 2 lines · Krishna Reddy

Good effort to isolate bus config from smmu drivers.

Thanks!

Quoted 3 lines

I have an orthogonal question here.
Can the following code handle the case, where different buses have different type of SMMU instances(like one bus has SMMUv2 and another bus has SMMUv3)? If it need to handle the above case, can the smmu device bus be matched with specific bus here and ops set only for that bus?

Not yet, but that is one of the end goals that this is all working towards. I think the stuff that I've added to the dev branch[1] today should have reached the point where that becomes viable, but I'll need to rig up a system to test it next week.

Intermediate solutions aren't worth it because in practice you inevitably end up needing both IOMMU drivers to share the platform "bus" anyway.

Cheers,
Robin.

[1] https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus

Quoted 19 lines
Quoted 13 lines
drivers/iommu/iommu.c +10 -0
+       for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
+               struct bus_type *bus = iommu_buses[i];
+               const struct iommu_ops *bus_ops = bus->iommu_ops;
+               int err;
+
+               WARN_ON(bus_ops && bus_ops != ops);
+               bus->iommu_ops = ops;
+               err = bus_iommu_probe(bus);
+               if (err) {
+                       bus_for_each_dev(bus, NULL, iommu,

remove_iommu_group);

drivers/iommu/iommu.c int iommu_device_register(struct iommu_device *iommu, · line 222

Quoted 4 lines
drivers/iommu/iommu.c +4 -0
+                       bus->iommu_ops = bus_ops;
+                       return err;
+               }
+       }
Quoted 1 line

-KR

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#15 lore raw

PATCH 3/13 iommu/amd: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and garbage-collect the last remnants of amd_iommu_init_api().

---
 drivers/iommu/amd/amd_iommu.h |  1 -
 drivers/iommu/amd/init.c      |  9 +--------
 drivers/iommu/amd/iommu.c     | 21 ---------------------
 3 files changed, 1 insertion(+), 30 deletions(-)
drivers/iommu/amd/amd_iommu.h +0 -1
index 1ab31074f5b3..384393ce57fb 100644
@@ -18,7 +18,6 @@ extern void amd_iommu_restart_event_logging(struct amd_iommu *iommu);
 extern int amd_iommu_init_devices(void);
 extern void amd_iommu_uninit_devices(void);
 extern void amd_iommu_init_notifier(void);
-extern int amd_iommu_init_api(void);
 
 #ifdef CONFIG_AMD_IOMMU_DEBUGFS
 void amd_iommu_debugfs_setup(struct amd_iommu *iommu);
drivers/iommu/amd/init.c +1 -8
index 0467918bf7fd..1cb10d8b0df4 100644
@@ -1970,20 +1970,13 @@ static int __init amd_iommu_init_pci(void)
 	/*
 	 * Order is important here to make sure any unity map requirements are
 	 * fulfilled. The unity mappings are created and written to the device
-	 * table during the amd_iommu_init_api() call.
+	 * table during the iommu_init_pci() call.
 	 *
 	 * After that we call init_device_table_dma() to make sure any
 	 * uninitialized DTE will block DMA, and in the end we flush the caches
 	 * of all IOMMUs to make sure the changes to the device table are
 	 * active.
 	 */
-	ret = amd_iommu_init_api();
-	if (ret) {
-		pr_err("IOMMU: Failed to initialize IOMMU-API interface (error=%d)!\n",
-		       ret);
-		goto out;
-	}
-
 	init_device_table_dma();
 
 	for_each_iommu(iommu)
drivers/iommu/amd/iommu.c +0 -21
index 6366a473ef0d..c0f8a541a7d6 100644
@@ -11,8 +11,6 @@
 #include <linux/ratelimit.h>
 #include <linux/pci.h>
 #include <linux/acpi.h>
-#include <linux/amba/bus.h>
-#include <linux/platform_device.h>
 #include <linux/pci-ats.h>
 #include <linux/bitmap.h>
 #include <linux/slab.h>
@@ -1838,25 +1836,6 @@ void amd_iommu_domain_update(struct protection_domain *domain)
 	amd_iommu_domain_flush_complete(domain);
 }
 
-int __init amd_iommu_init_api(void)
-{
-	int err;
-
-	err = bus_set_iommu(&pci_bus_type, &amd_iommu_ops);
-	if (err)
-		return err;
-#ifdef CONFIG_ARM_AMBA
-	err = bus_set_iommu(&amba_bustype, &amd_iommu_ops);
-	if (err)
-		return err;
-#endif
-	err = bus_set_iommu(&platform_bus_type, &amd_iommu_ops);
-	if (err)
-		return err;
-
-	return 0;
-}
-
 /*****************************************************************************
  *
  * The following functions belong to the exported interface of AMD IOMMU
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#16 lore raw

PATCH 4/13 iommu/arm-smmu: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary. With device probes now replayed for every IOMMU instance registration, the whole sorry ordering workaround for legacy DT bindings goes too, hooray!

---
 drivers/iommu/arm/arm-smmu/arm-smmu.c | 84 +--------------------------
 1 file changed, 2 insertions(+), 82 deletions(-)
drivers/iommu/arm/arm-smmu/arm-smmu.c +2 -82
index 52bd42d80b4f..34cab56b9c6d 100644
@@ -37,7 +37,6 @@
 #include <linux/ratelimit.h>
 #include <linux/slab.h>
 
-#include <linux/amba/bus.h>
 #include <linux/fsl/mc.h>
 
 #include "arm-smmu.h"
@@ -93,8 +92,6 @@ static struct platform_driver arm_smmu_driver;
 static struct iommu_ops arm_smmu_ops;
 
 #ifdef CONFIG_ARM_SMMU_LEGACY_DT_BINDINGS
-static int arm_smmu_bus_init(struct iommu_ops *ops);
-
 static struct device_node *dev_get_dev_node(struct device *dev)
 {
 	if (dev_is_pci(dev)) {
@@ -180,20 +177,6 @@ static int arm_smmu_register_legacy_master(struct device *dev,
 	kfree(sids);
 	return err;
 }
-
-/*
- * With the legacy DT binding in play, we have no guarantees about
- * probe order, but then we're also not doing default domains, so we can
- * delay setting bus ops until we're sure every possible SMMU is ready,
- * and that way ensure that no probe_device() calls get missed.
- */
-static int arm_smmu_legacy_bus_init(void)
-{
-	if (using_legacy_binding)
-		return arm_smmu_bus_init(&arm_smmu_ops);
-	return 0;
-}
-device_initcall_sync(arm_smmu_legacy_bus_init);
 #else
 static int arm_smmu_register_legacy_master(struct device *dev,
 					   struct arm_smmu_device **smmu)
@@ -2022,52 +2005,6 @@ static int arm_smmu_device_dt_probe(struct arm_smmu_device *smmu,
 	return 0;
 }
 
-static int arm_smmu_bus_init(struct iommu_ops *ops)
-{
-	int err;
-
-	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type)) {
-		err = bus_set_iommu(&platform_bus_type, ops);
-		if (err)
-			return err;
-	}
-#ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype)) {
-		err = bus_set_iommu(&amba_bustype, ops);
-		if (err)
-			goto err_reset_platform_ops;
-	}
-#endif
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		err = bus_set_iommu(&pci_bus_type, ops);
-		if (err)
-			goto err_reset_amba_ops;
-	}
-#endif
-#ifdef CONFIG_FSL_MC_BUS
-	if (!iommu_present(&fsl_mc_bus_type)) {
-		err = bus_set_iommu(&fsl_mc_bus_type, ops);
-		if (err)
-			goto err_reset_pci_ops;
-	}
-#endif
-	return 0;
-
-err_reset_pci_ops: __maybe_unused;
-#ifdef CONFIG_PCI
-	bus_set_iommu(&pci_bus_type, NULL);
-#endif
-err_reset_amba_ops: __maybe_unused;
-#ifdef CONFIG_ARM_AMBA
-	bus_set_iommu(&amba_bustype, NULL);
-#endif
-err_reset_platform_ops: __maybe_unused;
-	bus_set_iommu(&platform_bus_type, NULL);
-	return err;
-}
-
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	struct resource *res;
@@ -2185,7 +2122,8 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	err = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev);
 	if (err) {
 		dev_err(dev, "Failed to register iommu\n");
-		goto err_sysfs_remove;
+		iommu_device_sysfs_remove(&smmu->iommu);
+		return err;
 	}
 
 	platform_set_drvdata(pdev, smmu);
@@ -2203,24 +2141,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		pm_runtime_enable(dev);
 	}
 
-	/*
-	 * For ACPI and generic DT bindings, an SMMU will be probed before
-	 * any device which might need it, so we want the bus ops in place
-	 * ready to handle default domain setup as soon as any SMMU exists.
-	 */
-	if (!using_legacy_binding) {
-		err = arm_smmu_bus_init(&arm_smmu_ops);
-		if (err)
-			goto err_unregister_device;
-	}
-
 	return 0;
-
-err_unregister_device:
-	iommu_device_unregister(&smmu->iommu);
-err_sysfs_remove:
-	iommu_device_sysfs_remove(&smmu->iommu);
-	return err;
 }
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
@@ -2233,7 +2154,6 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
 		dev_notice(&pdev->dev, "disabling translation\n");
 
-	arm_smmu_bus_init(NULL);
 	iommu_device_unregister(&smmu->iommu);
 	iommu_device_sysfs_remove(&smmu->iommu);
 
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
maintainer Acked-by on v1 4/13
#17 lore raw

Re: [PATCH 04/13] iommu/arm-smmu: Clean up bus_set_iommu()

Quoted 3 lines · Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary. With device probes now replayed for every IOMMU instance registration, the whole sorry ordering workaround for legacy DT bindings goes too, hooray!

Ha, I hope you tested this!

Quoted 4 lines
---
 drivers/iommu/arm/arm-smmu/arm-smmu.c | 84 +--------------------------
 1 file changed, 2 insertions(+), 82 deletions(-)

Assuming it works,

Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#18 lore raw

Re: [PATCH 04/13] iommu/arm-smmu: Clean up bus_set_iommu()

Quoted 6 lines · Will Deacon
Quoted 3 lines · Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary. With device probes now replayed for every IOMMU instance registration, the whole sorry ordering workaround for legacy DT bindings goes too, hooray!

Ha, I hope you tested this!

Oh alright then, since it's you... :)

I've hacked up a Juno DT with the old bindings, and (after needing a while to remember that they're fundamentally incompatible with disable_bypass), can confirm that with my whole dev branch including this series applied, it boots and creates IOMMU groups as expected. I then made the mistake of trying without the branch to check whether the squawks from iommu_setup_dma_ops() were new or not, and... well... plain rc3 doesn't even boot on the same setup - it's somehow blowing up in the failure cleanup path of iommu_bus_init(), apparently calling iommu_release_device() on something where dev->iommu->iommu_dev is NULL, for reasons that are far from clear and I'm not sure I can really be bothered to debug further... :/

Cheers,
Robin.

Quoted 10 lines
Quoted 4 lines
---
  drivers/iommu/arm/arm-smmu/arm-smmu.c | 84 +--------------------------
  1 file changed, 2 insertions(+), 82 deletions(-)

Assuming it works,

Will

maintainer
#19 lore raw

Re: [PATCH 04/13] iommu/arm-smmu: Clean up bus_set_iommu()

Quoted 10 lines · Robin Murphy
Quoted 6 lines · Will Deacon
Quoted 3 lines · Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary. With device probes now replayed for every IOMMU instance registration, the whole sorry ordering workaround for legacy DT bindings goes too, hooray!

Ha, I hope you tested this!

Oh alright then, since it's you... :)

Quoted 10 lines

I've hacked up a Juno DT with the old bindings, and (after needing a while to remember that they're fundamentally incompatible with disable_bypass), can confirm that with my whole dev branch including this series applied, it boots and creates IOMMU groups as expected. I then made the mistake of trying without the branch to check whether the squawks from iommu_setup_dma_ops() were new or not, and... well... plain rc3 doesn't even boot on the same setup - it's somehow blowing up in the failure cleanup path of iommu_bus_init(), apparently calling iommu_release_device() on something where dev->iommu->iommu_dev is NULL, for reasons that are far from clear and I'm not sure I can really be bothered to debug further... :/

Great, so your series is a fix!

Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#20 lore raw

PATCH 5/13 iommu/arm-smmu-v3: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 53 +--------------------
 1 file changed, 2 insertions(+), 51 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -51
index 6699333fd17b..b221525c31b9 100644
@@ -28,8 +28,6 @@
 #include <linux/pci-ats.h>
 #include <linux/platform_device.h>
 
-#include <linux/amba/bus.h>
-
 #include "arm-smmu-v3.h"
 #include "../../iommu-sva-lib.h"
 
@@ -3698,43 +3696,6 @@ static unsigned long arm_smmu_resource_size(struct arm_smmu_device *smmu)
 		return SZ_128K;
 }
 
-static int arm_smmu_set_bus_ops(struct iommu_ops *ops)
-{
-	int err;
-
-#ifdef CONFIG_PCI
-	if (pci_bus_type.iommu_ops != ops) {
-		err = bus_set_iommu(&pci_bus_type, ops);
-		if (err)
-			return err;
-	}
-#endif
-#ifdef CONFIG_ARM_AMBA
-	if (amba_bustype.iommu_ops != ops) {
-		err = bus_set_iommu(&amba_bustype, ops);
-		if (err)
-			goto err_reset_pci_ops;
-	}
-#endif
-	if (platform_bus_type.iommu_ops != ops) {
-		err = bus_set_iommu(&platform_bus_type, ops);
-		if (err)
-			goto err_reset_amba_ops;
-	}
-
-	return 0;
-
-err_reset_amba_ops:
-#ifdef CONFIG_ARM_AMBA
-	bus_set_iommu(&amba_bustype, NULL);
-#endif
-err_reset_pci_ops: __maybe_unused;
-#ifdef CONFIG_PCI
-	bus_set_iommu(&pci_bus_type, NULL);
-#endif
-	return err;
-}
-
 static void __iomem *arm_smmu_ioremap(struct device *dev, resource_size_t start,
 				      resource_size_t size)
 {
@@ -3838,27 +3799,17 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	ret = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev);
 	if (ret) {
 		dev_err(dev, "Failed to register iommu\n");
-		goto err_sysfs_remove;
+		iommu_device_sysfs_remove(&smmu->iommu);
+		return ret;
 	}
 
-	ret = arm_smmu_set_bus_ops(&arm_smmu_ops);
-	if (ret)
-		goto err_unregister_device;
-
 	return 0;
-
-err_unregister_device:
-	iommu_device_unregister(&smmu->iommu);
-err_sysfs_remove:
-	iommu_device_sysfs_remove(&smmu->iommu);
-	return ret;
 }
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
 	struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
 
-	arm_smmu_set_bus_ops(NULL);
 	iommu_device_unregister(&smmu->iommu);
 	iommu_device_sysfs_remove(&smmu->iommu);
 	arm_smmu_device_disable(smmu);
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
maintainer Acked-by on v1 5/13
#21 lore raw

Re: [PATCH 05/13] iommu/arm-smmu-v3: Clean up bus_set_iommu()

Quoted 7 lines · Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 53 +--------------------
 1 file changed, 2 insertions(+), 51 deletions(-)

Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#22 lore raw

PATCH 6/13 iommu/dart: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
 drivers/iommu/apple-dart.c | 30 +-----------------------------
 1 file changed, 1 insertion(+), 29 deletions(-)
drivers/iommu/apple-dart.c +1 -29
index decafb07ad08..a679e4c02291 100644
@@ -823,27 +823,6 @@ static irqreturn_t apple_dart_irq(int irq, void *dev)
 	return IRQ_HANDLED;
 }
 
-static int apple_dart_set_bus_ops(const struct iommu_ops *ops)
-{
-	int ret;
-
-	if (!iommu_present(&platform_bus_type)) {
-		ret = bus_set_iommu(&platform_bus_type, ops);
-		if (ret)
-			return ret;
-	}
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		ret = bus_set_iommu(&pci_bus_type, ops);
-		if (ret) {
-			bus_set_iommu(&platform_bus_type, NULL);
-			return ret;
-		}
-	}
-#endif
-	return 0;
-}
-
 static int apple_dart_probe(struct platform_device *pdev)
 {
 	int ret;
@@ -899,14 +878,10 @@ static int apple_dart_probe(struct platform_device *pdev)
 
 	platform_set_drvdata(pdev, dart);
 
-	ret = apple_dart_set_bus_ops(&apple_dart_iommu_ops);
-	if (ret)
-		goto err_free_irq;
-
 	ret = iommu_device_sysfs_add(&dart->iommu, dev, NULL, "apple-dart.%s",
 				     dev_name(&pdev->dev));
 	if (ret)
-		goto err_remove_bus_ops;
+		goto err_free_irq;
 
 	ret = iommu_device_register(&dart->iommu, &apple_dart_iommu_ops, dev);
 	if (ret)
@@ -920,8 +895,6 @@ static int apple_dart_probe(struct platform_device *pdev)
 
 err_sysfs_remove:
 	iommu_device_sysfs_remove(&dart->iommu);
-err_remove_bus_ops:
-	apple_dart_set_bus_ops(NULL);
 err_free_irq:
 	free_irq(dart->irq, dart);
 err_clk_disable:
@@ -936,7 +909,6 @@ static int apple_dart_remove(struct platform_device *pdev)
 
 	apple_dart_hw_reset(dart);
 	free_irq(dart->irq, dart);
-	apple_dart_set_bus_ops(NULL);
 
 	iommu_device_unregister(&dart->iommu);
 	iommu_device_sysfs_remove(&dart->iommu);
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#24 lore raw

PATCH 7/13 iommu/exynos: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the init failure path accordingly.

---
 drivers/iommu/exynos-iommu.c | 9 ---------
 1 file changed, 9 deletions(-)
drivers/iommu/exynos-iommu.c +0 -9
index 71f2018e23fe..359b255b3924 100644
@@ -1356,16 +1356,7 @@ static int __init exynos_iommu_init(void)
 		goto err_zero_lv2;
 	}
 
-	ret = bus_set_iommu(&platform_bus_type, &exynos_iommu_ops);
-	if (ret) {
-		pr_err("%s: Failed to register exynos-iommu driver.\n",
-								__func__);
-		goto err_set_iommu;
-	}
-
 	return 0;
-err_set_iommu:
-	kmem_cache_free(lv2table_kmem_cache, zero_lv2_table);
 err_zero_lv2:
 	platform_driver_unregister(&exynos_sysmmu_driver);
 err_reg_driver:
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#25 lore raw

PATCH 8/13 iommu/ipmmu-vmsa: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary. This also leaves the custom initcall effectively doing nothing but register the driver, which no longer needs to happen early either, so convert it to builtin_platform_driver().

---
 drivers/iommu/ipmmu-vmsa.c | 35 +----------------------------------
 1 file changed, 1 insertion(+), 34 deletions(-)
drivers/iommu/ipmmu-vmsa.c +1 -34
index 8fdb84b3642b..2549d32f0ddd 100644
@@ -1090,11 +1090,6 @@ static int ipmmu_probe(struct platform_device *pdev)
 		ret = iommu_device_register(&mmu->iommu, &ipmmu_ops, &pdev->dev);
 		if (ret)
 			return ret;
-
-#if defined(CONFIG_IOMMU_DMA)
-		if (!iommu_present(&platform_bus_type))
-			bus_set_iommu(&platform_bus_type, &ipmmu_ops);
-#endif
 	}
 
 	/*
@@ -1168,32 +1163,4 @@ static struct platform_driver ipmmu_driver = {
 	.probe = ipmmu_probe,
 	.remove	= ipmmu_remove,
 };
-
-static int __init ipmmu_init(void)
-{
-	struct device_node *np;
-	static bool setup_done;
-	int ret;
-
-	if (setup_done)
-		return 0;
-
-	np = of_find_matching_node(NULL, ipmmu_of_ids);
-	if (!np)
-		return 0;
-
-	of_node_put(np);
-
-	ret = platform_driver_register(&ipmmu_driver);
-	if (ret < 0)
-		return ret;
-
-#if defined(CONFIG_ARM) && !defined(CONFIG_IOMMU_DMA)
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &ipmmu_ops);
-#endif
-
-	setup_done = true;
-	return 0;
-}
-subsys_initcall(ipmmu_init);
+builtin_platform_driver(ipmmu_driver);
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#26 lore raw

PATCH 9/13 iommu/mtk: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure paths accordingly.

---
 drivers/iommu/mtk_iommu.c    | 13 +------------
 drivers/iommu/mtk_iommu_v1.c | 13 +------------
 2 files changed, 2 insertions(+), 24 deletions(-)
drivers/iommu/mtk_iommu.c +1 -12
index 6fd75a60abd6..4278d9e032ad 100644
@@ -920,19 +920,11 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 	spin_lock_init(&data->tlb_lock);
 	list_add_tail(&data->list, &m4ulist);
 
-	if (!iommu_present(&platform_bus_type)) {
-		ret = bus_set_iommu(&platform_bus_type, &mtk_iommu_ops);
-		if (ret)
-			goto out_list_del;
-	}
-
 	ret = component_master_add_with_match(dev, &mtk_iommu_com_ops, match);
 	if (ret)
-		goto out_bus_set_null;
+		goto out_list_del;
 	return ret;
 
-out_bus_set_null:
-	bus_set_iommu(&platform_bus_type, NULL);
 out_list_del:
 	list_del(&data->list);
 	iommu_device_unregister(&data->iommu);
@@ -952,9 +944,6 @@ static int mtk_iommu_remove(struct platform_device *pdev)
 	iommu_device_sysfs_remove(&data->iommu);
 	iommu_device_unregister(&data->iommu);
 
-	if (iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, NULL);
-
 	clk_disable_unprepare(data->bclk);
 	device_link_remove(data->smicomm_dev, &pdev->dev);
 	pm_runtime_disable(&pdev->dev);
drivers/iommu/mtk_iommu_v1.c +1 -12
index ecff800656e6..7d17d6a21803 100644
@@ -660,19 +660,11 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 	if (ret)
 		goto out_sysfs_remove;
 
-	if (!iommu_present(&platform_bus_type)) {
-		ret = bus_set_iommu(&platform_bus_type,  &mtk_iommu_ops);
-		if (ret)
-			goto out_dev_unreg;
-	}
-
 	ret = component_master_add_with_match(dev, &mtk_iommu_com_ops, match);
 	if (ret)
-		goto out_bus_set_null;
+		goto out_dev_unreg;
 	return ret;
 
-out_bus_set_null:
-	bus_set_iommu(&platform_bus_type, NULL);
 out_dev_unreg:
 	iommu_device_unregister(&data->iommu);
 out_sysfs_remove:
@@ -687,9 +679,6 @@ static int mtk_iommu_remove(struct platform_device *pdev)
 	iommu_device_sysfs_remove(&data->iommu);
 	iommu_device_unregister(&data->iommu);
 
-	if (iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, NULL);
-
 	clk_disable_unprepare(data->bclk);
 	devm_free_irq(&pdev->dev, data->irq, data);
 	component_master_del(&pdev->dev, &mtk_iommu_com_ops);
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#27 lore raw

PATCH 10/13 iommu/omap: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the init failure path accordingly.

---
 drivers/iommu/omap-iommu.c | 6 ------
 1 file changed, 6 deletions(-)
drivers/iommu/omap-iommu.c +0 -6
index d9cf2820c02e..07ee2600113c 100644
@@ -1776,14 +1776,8 @@ static int __init omap_iommu_init(void)
 		goto fail_driver;
 	}
 
-	ret = bus_set_iommu(&platform_bus_type, &omap_iommu_ops);
-	if (ret)
-		goto fail_bus;
-
 	return 0;
 
-fail_bus:
-	platform_driver_unregister(&omap_iommu_driver);
 fail_driver:
 	kmem_cache_destroy(iopte_cachep);
 	return ret;
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#28 lore raw

PATCH 11/13 iommu/tegra-smmu: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
 drivers/iommu/tegra-smmu.c | 29 ++++++-----------------------
 1 file changed, 6 insertions(+), 23 deletions(-)
drivers/iommu/tegra-smmu.c +6 -23
index 1fea68e551f1..2e4d2e4c65bb 100644
@@ -1086,8 +1086,8 @@ struct tegra_smmu *tegra_smmu_probe(struct device *dev,
 
 	/*
 	 * This is a bit of a hack. Ideally we'd want to simply return this
-	 * value. However the IOMMU registration process will attempt to add
-	 * all devices to the IOMMU when bus_set_iommu() is called. In order
+	 * value. However iommu_device_register() will attempt to add
+	 * all devices to the IOMMU before we get that far. In order
 	 * not to rely on global variables to track the IOMMU instance, we
 	 * set it here so that it can be looked up from the .probe_device()
 	 * callback via the IOMMU device's .drvdata field.
@@ -1141,32 +1141,15 @@ struct tegra_smmu *tegra_smmu_probe(struct device *dev,
 		return ERR_PTR(err);
 
 	err = iommu_device_register(&smmu->iommu, &tegra_smmu_ops, dev);
-	if (err)
-		goto remove_sysfs;
-
-	err = bus_set_iommu(&platform_bus_type, &tegra_smmu_ops);
-	if (err < 0)
-		goto unregister;
-
-#ifdef CONFIG_PCI
-	err = bus_set_iommu(&pci_bus_type, &tegra_smmu_ops);
-	if (err < 0)
-		goto unset_platform_bus;
-#endif
+	if (err) {
+		iommu_device_sysfs_remove(&smmu->iommu);
+		return ERR_PTR(err);
+	}
 
 	if (IS_ENABLED(CONFIG_DEBUG_FS))
 		tegra_smmu_debugfs_init(smmu);
 
 	return smmu;
-
-unset_platform_bus: __maybe_unused;
-	bus_set_iommu(&platform_bus_type, NULL);
-unregister:
-	iommu_device_unregister(&smmu->iommu);
-remove_sysfs:
-	iommu_device_sysfs_remove(&smmu->iommu);
-
-	return ERR_PTR(err);
 }
 
 void tegra_smmu_remove(struct tegra_smmu *smmu)
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author reviewer
#29 lore raw

PATCH 12/13 iommu/virtio: Clean up bus_set_iommu()

In reply to Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
 drivers/iommu/virtio-iommu.c | 24 ------------------------
 1 file changed, 24 deletions(-)
drivers/iommu/virtio-iommu.c +0 -24
index 25be4b822aa0..371f8657c0ce 100644
@@ -7,7 +7,6 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
-#include <linux/amba/bus.h>
 #include <linux/delay.h>
 #include <linux/dma-iommu.h>
 #include <linux/dma-map-ops.h>
@@ -1146,26 +1145,6 @@ static int viommu_probe(struct virtio_device *vdev)
 
 	iommu_device_register(&viommu->iommu, &viommu_ops, parent_dev);
 
-#ifdef CONFIG_PCI
-	if (pci_bus_type.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&pci_bus_type, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-#endif
-#ifdef CONFIG_ARM_AMBA
-	if (amba_bustype.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&amba_bustype, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-#endif
-	if (platform_bus_type.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&platform_bus_type, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-
 	vdev->priv = viommu;
 
 	dev_info(dev, "input address: %u bits\n",
@@ -1174,9 +1153,6 @@ static int viommu_probe(struct virtio_device *vdev)
 
 	return 0;
 
-err_unregister:
-	iommu_device_sysfs_remove(&viommu->iommu);
-	iommu_device_unregister(&viommu->iommu);
 err_free_vqs:
 	vdev->config->del_vqs(vdev);
 
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Reviewed-by on v1 12/13
#30 lore raw

Re: [PATCH 12/13] iommu/virtio: Clean up bus_set_iommu()

Quoted 13 lines · Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
 drivers/iommu/virtio-iommu.c | 24 ------------------------
 1 file changed, 24 deletions(-)
drivers/iommu/virtio-iommu.c +0 -0
index 25be4b822aa0..371f8657c0ce 100644

drivers/iommu/virtio-iommu.c line 12

Quoted 7 lines
drivers/iommu/virtio-iommu.c +0 -1
@@ -7,7 +7,6 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
-#include <linux/amba/bus.h>
 #include <linux/delay.h>
 #include <linux/dma-iommu.h>
 #include <linux/dma-map-ops.h>

<linux/platform_device.h> isn't needed anymore either. In any case it looks great, thanks

and tested on QEMU (so only PCI for now)

Quoted 40 lines
drivers/iommu/virtio-iommu.c +0 -23
@@ -1146,26 +1145,6 @@ static int viommu_probe(struct virtio_device *vdev)
 
 	iommu_device_register(&viommu->iommu, &viommu_ops, parent_dev);
 
-#ifdef CONFIG_PCI
-	if (pci_bus_type.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&pci_bus_type, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-#endif
-#ifdef CONFIG_ARM_AMBA
-	if (amba_bustype.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&amba_bustype, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-#endif
-	if (platform_bus_type.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&platform_bus_type, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-
 	vdev->priv = viommu;
 
 	dev_info(dev, "input address: %u bits\n",
@@ -1174,9 +1153,6 @@ static int viommu_probe(struct virtio_device *vdev)
 
 	return 0;
 
-err_unregister:
-	iommu_device_sysfs_remove(&viommu->iommu);
-	iommu_device_unregister(&viommu->iommu);
 err_free_vqs:
 	vdev->config->del_vqs(vdev);
 
Signature · 1 line
2.28.0.dirty
author reviewer
#31 lore raw

Re: [PATCH 12/13] iommu/virtio: Clean up bus_set_iommu()

Quoted 17 lines · Jean-Philippe Brucker
Quoted 16 lines · Robin Murphy

Stop calling bus_set_iommu() since it's now unnecessary, and simplify the probe failure path accordingly.

---
  drivers/iommu/virtio-iommu.c | 24 ------------------------
  1 file changed, 24 deletions(-)
drivers/iommu/virtio-iommu.c +0 -0
index 25be4b822aa0..371f8657c0ce 100644
@@ -7,7 +7,6 @@
 
  #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
Quoted 7 lines

drivers/iommu/virtio-iommu.c line 12

Quoted 4 lines
drivers/iommu/virtio-iommu.c +0 -1
-#include <linux/amba/bus.h>
  #include <linux/delay.h>
  #include <linux/dma-iommu.h>
  #include <linux/dma-map-ops.h>

<linux/platform_device.h> isn't needed anymore either. In any case it looks great, thanks

Ha, it totally passed me by that this one *isn't* a platform driver, derp :)

Quoted 3 lines

and tested on QEMU (so only PCI for now)

Thanks!

Robin.

Quoted 37 lines
Quoted 37 lines
drivers/iommu/virtio-iommu.c +0 -23
@@ -1146,26 +1145,6 @@ static int viommu_probe(struct virtio_device *vdev)
 
  	iommu_device_register(&viommu->iommu, &viommu_ops, parent_dev);
 
-#ifdef CONFIG_PCI
-	if (pci_bus_type.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&pci_bus_type, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-#endif
-#ifdef CONFIG_ARM_AMBA
-	if (amba_bustype.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&amba_bustype, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-#endif
-	if (platform_bus_type.iommu_ops != &viommu_ops) {
-		ret = bus_set_iommu(&platform_bus_type, &viommu_ops);
-		if (ret)
-			goto err_unregister;
-	}
-
  	vdev->priv = viommu;
 
  	dev_info(dev, "input address: %u bits\n",
@@ -1174,9 +1153,6 @@ static int viommu_probe(struct virtio_device *vdev)
 
  	return 0;
 
-err_unregister:
-	iommu_device_sysfs_remove(&viommu->iommu);
-	iommu_device_unregister(&viommu->iommu);
  err_free_vqs:
  	vdev->config->del_vqs(vdev);
 
Quoted 3 lines
Quoted 3 lines
Signature · 1 line
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author reviewer
#32 lore raw

PATCH 13/13 iommu: Clean up bus_set_iommu()

In reply to Robin Murphy

Clean up the remaining trivial bus_set_iommu() callsites along with the implementation. Now drivers only have to know and care about iommu_device instances, phew!

---
 drivers/iommu/arm/arm-smmu/qcom_iommu.c |  4 ----
 drivers/iommu/fsl_pamu_domain.c         |  4 ----
 drivers/iommu/intel/iommu.c             |  1 -
 drivers/iommu/iommu.c                   | 24 ------------------------
 drivers/iommu/msm_iommu.c               |  2 --
 drivers/iommu/rockchip-iommu.c          |  2 --
 drivers/iommu/s390-iommu.c              |  6 ------
 drivers/iommu/sprd-iommu.c              |  5 -----
 drivers/iommu/sun50i-iommu.c            |  2 --
 include/linux/iommu.h                   |  1 -
 10 files changed, 51 deletions(-)
drivers/iommu/arm/arm-smmu/qcom_iommu.c +0 -4
index 8cd39abade5a..028649203d33 100644
@@ -845,8 +845,6 @@ static int qcom_iommu_device_probe(struct platform_device *pdev)
 		goto err_pm_disable;
 	}
 
-	bus_set_iommu(&platform_bus_type, &qcom_iommu_ops);
-
 	if (qcom_iommu->local_base) {
 		pm_runtime_get_sync(dev);
 		writel_relaxed(0xffffffff, qcom_iommu->local_base + SMMU_INTR_SEL_NS);
@@ -864,8 +862,6 @@ static int qcom_iommu_device_remove(struct platform_device *pdev)
 {
 	struct qcom_iommu_dev *qcom_iommu = platform_get_drvdata(pdev);
 
-	bus_set_iommu(&platform_bus_type, NULL);
-
 	pm_runtime_force_suspend(&pdev->dev);
 	platform_set_drvdata(pdev, NULL);
 	iommu_device_sysfs_remove(&qcom_iommu->iommu);
drivers/iommu/fsl_pamu_domain.c +0 -4
index d03a14386f86..ddf5ab28615c 100644
@@ -480,11 +480,7 @@ int __init pamu_domain_init(void)
 	if (ret) {
 		iommu_device_sysfs_remove(&pamu_iommu);
 		pr_err("Can't register iommu device\n");
-		return ret;
 	}
 
-	bus_set_iommu(&platform_bus_type, &fsl_pamu_ops);
-	bus_set_iommu(&pci_bus_type, &fsl_pamu_ops);
-
 	return ret;
 }
drivers/iommu/intel/iommu.c +0 -1
index 0b24c4977dbe..49d552a96098 100644
@@ -4178,7 +4178,6 @@ int __init intel_iommu_init(void)
 	}
 	up_read(&dmar_global_lock);
 
-	bus_set_iommu(&pci_bus_type, &intel_iommu_ops);
 	if (si_domain && !hw_pass_through)
 		register_memory_notifier(&intel_iommu_memory_nb);
 
drivers/iommu/iommu.c +0 -24
index 51205c33c426..7800e342d285 100644
@@ -1885,30 +1885,6 @@ static int iommu_bus_init(struct bus_type *bus)
 	return err;
 }
 
-/**
- * bus_set_iommu - set iommu-callbacks for the bus
- * @bus: bus.
- * @ops: the callbacks provided by the iommu-driver
- *
- * This function is called by an iommu driver to set the iommu methods
- * used for a particular bus. Drivers for devices on that bus can use
- * the iommu-api after these ops are registered.
- * This special function is needed because IOMMUs are usually devices on
- * the bus itself, so the iommu drivers are not initialized when the bus
- * is set up. With this function the iommu-driver can set the iommu-ops
- * afterwards.
- */
-int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
-{
-	if (bus->iommu_ops && ops && bus->iommu_ops != ops)
-		return -EBUSY;
-
-	bus->iommu_ops = ops;
-
-	return 0;
-}
-EXPORT_SYMBOL_GPL(bus_set_iommu);
-
 bool iommu_present(struct bus_type *bus)
 {
 	return bus->iommu_ops != NULL;
drivers/iommu/msm_iommu.c +0 -2
index 50f57624610f..5b89fb16feb8 100644
@@ -789,8 +789,6 @@ static int msm_iommu_probe(struct platform_device *pdev)
 		goto fail;
 	}
 
-	bus_set_iommu(&platform_bus_type, &msm_iommu_ops);
-
 	pr_info("device mapped at %p, irq %d with %d ctx banks\n",
 		iommu->base, iommu->irq, iommu->ncb);
 
drivers/iommu/rockchip-iommu.c +0 -2
index ab57c4b8fade..a3fc59b814ab 100644
@@ -1300,8 +1300,6 @@ static int rk_iommu_probe(struct platform_device *pdev)
 	if (!dma_dev)
 		dma_dev = &pdev->dev;
 
-	bus_set_iommu(&platform_bus_type, &rk_iommu_ops);
-
 	pm_runtime_enable(dev);
 
 	for (i = 0; i < iommu->num_irq; i++) {
drivers/iommu/s390-iommu.c +0 -6
index bd56e21bf754..ea4ba9de04af 100644
@@ -376,9 +376,3 @@ static const struct iommu_ops s390_iommu_ops = {
 		.free		= s390_domain_free,
 	}
 };
-
-static int __init s390_iommu_init(void)
-{
-	return bus_set_iommu(&pci_bus_type, &s390_iommu_ops);
-}
-subsys_initcall(s390_iommu_init);
drivers/iommu/sprd-iommu.c +0 -5
index bd409bab6286..6770e6a72283 100644
@@ -507,9 +507,6 @@ static int sprd_iommu_probe(struct platform_device *pdev)
 	if (ret)
 		goto remove_sysfs;
 
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &sprd_iommu_ops);
-
 	ret = sprd_iommu_clk_enable(sdev);
 	if (ret)
 		goto unregister_iommu;
@@ -545,8 +542,6 @@ static int sprd_iommu_remove(struct platform_device *pdev)
 	iommu_group_put(sdev->group);
 	sdev->group = NULL;
 
-	bus_set_iommu(&platform_bus_type, NULL);
-
 	platform_set_drvdata(pdev, NULL);
 	iommu_device_sysfs_remove(&sdev->iommu);
 	iommu_device_unregister(&sdev->iommu);
drivers/iommu/sun50i-iommu.c +0 -2
index c54ab477b8fd..e104543b78d9 100644
@@ -968,8 +968,6 @@ static int sun50i_iommu_probe(struct platform_device *pdev)
 	if (ret < 0)
 		goto err_unregister;
 
-	bus_set_iommu(&platform_bus_type, &sun50i_iommu_ops);
-
 	return 0;
 
 err_unregister:
include/linux/iommu.h +0 -1
index 46d9e27d49fd..e93e4a42e1a0 100644
@@ -416,7 +416,6 @@ static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
 #define IOMMU_GROUP_NOTIFY_UNBIND_DRIVER	5 /* Pre Driver unbind */
 #define IOMMU_GROUP_NOTIFY_UNBOUND_DRIVER	6 /* Post Driver unbind */
 
-extern int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops);
 extern int bus_iommu_probe(struct bus_type *bus);
 extern bool iommu_present(struct bus_type *bus);
 extern bool device_iommu_capable(struct device *dev, enum iommu_cap cap);
Signature · 6 lines
2.28.0.dirty

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
maintainer Tested-by on the v1 series
#33 lore raw
In reply to Robin Murphy

Hi Robin,

Quoted 20 lines · Robin Murphy

Hi all,

Here's another chapter in my saga of moving to per-instance IOMMU ops - iommu_present() and iommu_capable() cleanups will be ongoing for another cycle or two, while this one is at least self-contained within the subsystem. The next steps after this are making iommu_domain_alloc() instance-aware - which should finish the public API - and pulling the fwnode/of_xlate bits into __iommu_probe_device(). And then making sense of whatever's left :)

For ease of review here I split out individual driver patches based on whether there was any non-trivial change or affect on control flow; the straightforward deletions are all lumped together since the whole series needs applying together either way, but I'm happy to split the final patch up further if anyone would like.

Patch #3 for AMD is based on Mario's SWIOTLB patch here:

https://lore.kernel.org/linux-iommu/[email protected]/

Quoted 8 lines

since that wants merging first as fix material. The series is also based contextually (but not functionally) on my device_iommu_capable() patches here:

https://lore.kernel.org/linux-iommu/[email protected]/

since those are pretty much good to go now (I'll send a slightly-tweaked final version once the iommu/core branch is open).

Works fine on Samsung Exynos based boards (both, ARM 32bit and 64bit).

relevant changes

Quoted 36 lines

Thanks,
Robin.

Robin Murphy (13):
iommu: Always register bus notifiers
iommu: Move bus setup to IOMMU device registration iommu/amd: Clean up bus_set_iommu() iommu/arm-smmu: Clean up bus_set_iommu() iommu/arm-smmu-v3: Clean up bus_set_iommu() iommu/dart: Clean up bus_set_iommu() iommu/exynos: Clean up bus_set_iommu() iommu/ipmmu-vmsa: Clean up bus_set_iommu() iommu/mtk: Clean up bus_set_iommu() iommu/omap: Clean up bus_set_iommu() iommu/tegra-smmu: Clean up bus_set_iommu() iommu/virtio: Clean up bus_set_iommu()
iommu: Clean up bus_set_iommu()

  drivers/iommu/amd/amd_iommu.h               |   1 -
  drivers/iommu/amd/init.c                    |   9 +-
  drivers/iommu/amd/iommu.c                   |  21 ----
  drivers/iommu/apple-dart.c                  |  30 +-----
  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  53 +---------
  drivers/iommu/arm/arm-smmu/arm-smmu.c       |  84 +--------------
  drivers/iommu/arm/arm-smmu/qcom_iommu.c     |   4 -
  drivers/iommu/exynos-iommu.c                |   9 --
  drivers/iommu/fsl_pamu_domain.c             |   4 -
  drivers/iommu/intel/iommu.c                 |   1 -
  drivers/iommu/iommu.c                       | 109 +++++++++-----------
  drivers/iommu/ipmmu-vmsa.c                  |  35 +------
  drivers/iommu/msm_iommu.c                   |   2 -
  drivers/iommu/mtk_iommu.c                   |  13 +--
  drivers/iommu/mtk_iommu_v1.c                |  13 +--
  drivers/iommu/omap-iommu.c                  |   6 --
Quoted 9 lines
  drivers/iommu/rockchip-iommu.c              |   2 -
  drivers/iommu/s390-iommu.c                  |   6 --
  drivers/iommu/sprd-iommu.c                  |   5 -
  drivers/iommu/sun50i-iommu.c                |   2 -
  drivers/iommu/tegra-smmu.c                  |  29 ++----
  drivers/iommu/virtio-iommu.c                |  24 -----
  include/linux/iommu.h                       |   1 -
  23 files changed, 62 insertions(+), 401 deletions(-)

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland