Skip to contentSkip to conversation

drivers: iommu: Use of add_device()/remove_device() outside of IOMMU groups

PATCH In review Damian Hobson-Garcia 3 participants 6 messages Jan 15, 2013 – Jan 18, 2013 4 messages not in the archive IOMMU

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drivers: iommu: Use of add_device()/remove_device() outside of IOMMU groups

We've recently been thinking about the best way to attach platform devices in
an IOMMU in a generic way without having to have every platform device
register itself at probe time. We've seen that at least on the tegra IOMMU,
using the platform_bus notifier seems like a viable option (http://lists.linuxfoundation.org/pipermail/iommu/2012-December/004954.html). We also noticed that the IOMMU ops add_device() function is called (indirectly) from a similar bus
notifier and we're thinking that using the bus notifier in the common code
would be better than us rolling our own to do basically the same thing.

The problem, however, is that add_device() seems to be intended for use with IOMMU groups. i.e. After calling add_device() on a device that is should be in a group, dev->iommu_group should have some meaningful value.
I think though, that if the IOMMU groups requirement can be removed, it should be more versatile.

I believe that removing the IOMMU groups dependency can be achieved with the patch below. Currently the intel-iommu seems to be the only implementation using these callbacks.

---
 drivers/iommu/intel-iommu.c |    3 ++-
 drivers/iommu/iommu.c       |    2 +-
 include/linux/iommu.h       |    4 ++--
 3 files changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +2 -1
index b9d0911..a763872 100644
@@ -4217,7 +4217,8 @@ root_bus:
 
 static void intel_iommu_remove_device(struct device *dev)
 {
-	iommu_group_remove_device(dev);
+	if (dev->iommu_group)
+		iommu_group_remove_device(dev);
 }
 
 static struct iommu_ops intel_iommu_ops = {
drivers/iommu/iommu.c +1 -1
index ddbdaca..a6046c2 100644
@@ -510,7 +510,7 @@ static int iommu_bus_notifier(struct notifier_block *nb,
 		if (ops->add_device)
 			return ops->add_device(dev);
 	} else if (action == BUS_NOTIFY_DEL_DEVICE) {
-		if (ops->remove_device && dev->iommu_group) {
+		if (ops->remove_device) {
 			ops->remove_device(dev);
 			return 0;
 		}
include/linux/iommu.h +2 -2
index f3b99e1..2715592 100644
@@ -74,8 +74,8 @@ enum iommu_attr {
  * @unmap: unmap a physically contiguous memory region from an iommu domain
  * @iova_to_phys: translate iova to physical address
  * @domain_has_cap: domain capabilities query
- * @add_device: add device to iommu grouping
- * @remove_device: remove device from iommu grouping
+ * @add_device: add device to iommu grouping or domain
+ * @remove_device: remove device from iommu grouping or domain
  * @domain_get_attr: Query domain attributes
  * @domain_set_attr: Change domain attributes
  * @pgsize_bitmap: bitmap of supported page sizes
Signature · 1 line
1.7.5.4

A message that is not in the archive (sent off-list or not yet received): 1358244652-12728-1-git-send-email-dhobsong-AlSX/UN32fvPDbFq/[email protected]

#2 lore raw
Quoted 14 lines · Damian Hobson-Garcia

We've recently been thinking about the best way to attach platform devices in
an IOMMU in a generic way without having to have every platform device
register itself at probe time. We've seen that at least on the tegra IOMMU,
using the platform_bus notifier seems like a viable option (http://lists.linuxfoundation.org/pipermail/iommu/2012-December/004954.html). We also noticed that the IOMMU ops add_device() function is called (indirectly) from a similar bus
notifier and we're thinking that using the bus notifier in the common code
would be better than us rolling our own to do basically the same thing.

The problem, however, is that add_device() seems to be intended for use with IOMMU groups. i.e. After calling add_device() on a device that is should be in a group, dev->iommu_group should have some meaningful value.
I think though, that if the IOMMU groups requirement can be removed, it should be more versatile.

Quoted 3 lines

I believe that removing the IOMMU groups dependency can be achieved with the patch below. Currently the intel-iommu seems to be the only implementation using these callbacks.

What stops you from creating a single iommu_group and adding all device to it? Thanks,

Alex

Quoted 49 lines
---
 drivers/iommu/intel-iommu.c |    3 ++-
 drivers/iommu/iommu.c       |    2 +-
 include/linux/iommu.h       |    4 ++--
 3 files changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +2 -1
index b9d0911..a763872 100644
@@ -4217,7 +4217,8 @@ root_bus:
 
 static void intel_iommu_remove_device(struct device *dev)
 {
-	iommu_group_remove_device(dev);
+	if (dev->iommu_group)
+		iommu_group_remove_device(dev);
 }
 
 static struct iommu_ops intel_iommu_ops = {
drivers/iommu/iommu.c +1 -1
index ddbdaca..a6046c2 100644
@@ -510,7 +510,7 @@ static int iommu_bus_notifier(struct notifier_block *nb,
 		if (ops->add_device)
 			return ops->add_device(dev);
 	} else if (action == BUS_NOTIFY_DEL_DEVICE) {
-		if (ops->remove_device && dev->iommu_group) {
+		if (ops->remove_device) {
 			ops->remove_device(dev);
 			return 0;
 		}
include/linux/iommu.h +2 -2
index f3b99e1..2715592 100644
@@ -74,8 +74,8 @@ enum iommu_attr {
  * @unmap: unmap a physically contiguous memory region from an iommu domain
  * @iova_to_phys: translate iova to physical address
  * @domain_has_cap: domain capabilities query
- * @add_device: add device to iommu grouping
- * @remove_device: remove device from iommu grouping
+ * @add_device: add device to iommu grouping or domain
+ * @remove_device: remove device from iommu grouping or domain
  * @domain_get_attr: Query domain attributes
  * @domain_set_attr: Change domain attributes
  * @pgsize_bitmap: bitmap of supported page sizes

A message that is not in the archive (sent off-list or not yet received): 1358372869.11144.21.camel-xdHQ/5r00wBBDLzU/[email protected]

#4 lore raw

I mistakenly sent this with the tag [PATCH] when it should have been [RFC]. Updated with this reply.

Hi Alex,

Quoted 15 lines · Alex Williamson
Quoted 14 lines · Damian Hobson-Garcia

We've recently been thinking about the best way to attach platform devices in
an IOMMU in a generic way without having to have every platform device
register itself at probe time. We've seen that at least on the tegra IOMMU,
using the platform_bus notifier seems like a viable option (http://lists.linuxfoundation.org/pipermail/iommu/2012-December/004954.html). We also noticed that the IOMMU ops add_device() function is called (indirectly) from a similar bus
notifier and we're thinking that using the bus notifier in the common code
would be better than us rolling our own to do basically the same thing.

The problem, however, is that add_device() seems to be intended for use with IOMMU groups. i.e. After calling add_device() on a device that is should be in a group, dev->iommu_group should have some meaningful value.
I think though, that if the IOMMU groups requirement can be removed, it should be more versatile.

Quoted 8 lines
Quoted 3 lines

I believe that removing the IOMMU groups dependency can be achieved with the patch below. Currently the intel-iommu seems to be the only implementation using these callbacks.

What stops you from creating a single iommu_group and adding all device to it? Thanks,

Alex

Thanks for your comment.

Nothing stops me, but putting everything in one group doesn't really add any benefit over not using groups at all, does it? It seems like its just adding more complexity(i.e code) to the driver without adding any additional functionality.

Thanks,
Damian

Quoted 53 lines
Quoted 50 lines
---
  drivers/iommu/intel-iommu.c |    3 ++-
  drivers/iommu/iommu.c       |    2 +-
  include/linux/iommu.h       |    4 ++--
  3 files changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +2 -1
index b9d0911..a763872 100644
@@ -4217,7 +4217,8 @@ root_bus:
 
  static void intel_iommu_remove_device(struct device *dev)
  {
-	iommu_group_remove_device(dev);
+	if (dev->iommu_group)
+		iommu_group_remove_device(dev);
  }
 
  static struct iommu_ops intel_iommu_ops = {
drivers/iommu/iommu.c +1 -1
index ddbdaca..a6046c2 100644
@@ -510,7 +510,7 @@ static int iommu_bus_notifier(struct notifier_block *nb,
  		if (ops->add_device)
  			return ops->add_device(dev);
  	} else if (action == BUS_NOTIFY_DEL_DEVICE) {
-		if (ops->remove_device && dev->iommu_group) {
+		if (ops->remove_device) {
  			ops->remove_device(dev);
  			return 0;
  		}
include/linux/iommu.h +2 -2
index f3b99e1..2715592 100644
@@ -74,8 +74,8 @@ enum iommu_attr {
   * @unmap: unmap a physically contiguous memory region from an iommu domain
   * @iova_to_phys: translate iova to physical address
   * @domain_has_cap: domain capabilities query
- * @add_device: add device to iommu grouping
- * @remove_device: remove device from iommu grouping
+ * @add_device: add device to iommu grouping or domain
+ * @remove_device: remove device from iommu grouping or domain
   * @domain_get_attr: Query domain attributes
   * @domain_set_attr: Change domain attributes
   * @pgsize_bitmap: bitmap of supported page sizes
Signature · 3 lines
Damian Hobson-Garcia
IGEL Co.,Ltd
http://www.igel.co.jp

A message that is not in the archive (sent off-list or not yet received): 50F76E12.7020607-AlSX/UN32fvPDbFq/[email protected]

#6 lore raw
Quoted 43 lines

-----Original Message-----
From: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org [mailto:iommu- bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org] On Behalf Of Damian Hobson-Garcia
Sent: Thursday, January 17, 2013 8:51 AM
To: Alex Williamson
Cc: hdk-AlSX/UN32fvPDbFq/[email protected]; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; [email protected]; laurent.pinchart-ryLnwIuWjnjg/[email protected]
Subject: Re: [RFC] drivers: iommu: Use of add_device()/remove_device()
outside of IOMMU groups

I mistakenly sent this with the tag [PATCH] when it should have been [RFC]. Updated with this reply.

Hi Alex,

Quoted 8 lines · Alex Williamson
Quoted 7 lines · Damian Hobson-Garcia

We've recently been thinking about the best way to attach platform devices in an IOMMU in a generic way without having to have every platform device register itself at probe time. We've seen that at least on the tegra IOMMU, using the platform_bus notifier seems like a viable option
(http://lists.linuxfoundation.org/pipermail/iommu/2012-December/00495
4.html). We also noticed that the IOMMU ops add_device() function is

called (indirectly) from a similar bus notifier and we're thinking that using the bus notifier in the common code would be better than us rolling our own to do basically the same thing.

Quoted 4 lines
Quoted 4 lines

The problem, however, is that add_device() seems to be intended for use with IOMMU groups. i.e. After calling add_device() on a device that is should be in a group, dev->iommu_group should have some

meaningful value.

Quoted 10 lines
Quoted 6 lines

I think though, that if the IOMMU groups requirement can be removed, it should be more versatile.

I believe that removing the IOMMU groups dependency can be achieved with the patch below. Currently the intel-iommu seems to be the only implementation using these callbacks.

What stops you from creating a single iommu_group and adding all device to it? Thanks,

Quoted 8 lines
Quoted 1 line

Alex

Thanks for your comment.

Nothing stops me, but putting everything in one group doesn't really add any benefit over not using groups at all, does it? It seems like its just adding more complexity(i.e code) to the driver without adding any additional functionality.

Do you intend to use vfio? That's the specific objective of creating iommu groups.

-Varun

#7 lore raw
In reply to a message not in the archive
Quoted 27 lines · Damian Hobson-Garcia

I mistakenly sent this with the tag [PATCH] when it should have been [RFC]. Updated with this reply.

Hi Alex,

Quoted 22 lines · Alex Williamson
Quoted 14 lines · Damian Hobson-Garcia

We've recently been thinking about the best way to attach platform devices in
an IOMMU in a generic way without having to have every platform device
register itself at probe time. We've seen that at least on the tegra IOMMU,
using the platform_bus notifier seems like a viable option (http://lists.linuxfoundation.org/pipermail/iommu/2012-December/004954.html). We also noticed that the IOMMU ops add_device() function is called (indirectly) from a similar bus
notifier and we're thinking that using the bus notifier in the common code
would be better than us rolling our own to do basically the same thing.

The problem, however, is that add_device() seems to be intended for use with IOMMU groups. i.e. After calling add_device() on a device that is should be in a group, dev->iommu_group should have some meaningful value.
I think though, that if the IOMMU groups requirement can be removed, it should be more versatile.

Quoted 3 lines

I believe that removing the IOMMU groups dependency can be achieved with the patch below. Currently the intel-iommu seems to be the only implementation using these callbacks.

What stops you from creating a single iommu_group and adding all device to it? Thanks,

Quoted 7 lines
Quoted 1 line

Alex

Thanks for your comment.

Nothing stops me, but putting everything in one group doesn't really add any benefit over not using groups at all, does it? It seems like its just adding more complexity(i.e code) to the driver without adding any additional functionality.

Does your iommu provide any isolation or is it translate-only. As Varun notes, groups enable vfio, so if there's any possibility that you care about userspace drivers for your devices or assigning devices to qemu guests, then groups might be useful. The overhead of groups is really meant to be very minimal for the iommu driver. That said, I guess I have no real objection to generalizing the iommu bus notifier for add/remove. Thanks,

Alex

Quoted 54 lines
Quoted 52 lines
Quoted 49 lines
---
  drivers/iommu/intel-iommu.c |    3 ++-
  drivers/iommu/iommu.c       |    2 +-
  include/linux/iommu.h       |    4 ++--
  3 files changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +2 -1
index b9d0911..a763872 100644
@@ -4217,7 +4217,8 @@ root_bus:
 
  static void intel_iommu_remove_device(struct device *dev)
  {
-	iommu_group_remove_device(dev);
+	if (dev->iommu_group)
+		iommu_group_remove_device(dev);
  }
 
  static struct iommu_ops intel_iommu_ops = {
drivers/iommu/iommu.c +1 -1
index ddbdaca..a6046c2 100644
@@ -510,7 +510,7 @@ static int iommu_bus_notifier(struct notifier_block *nb,
  		if (ops->add_device)
  			return ops->add_device(dev);
  	} else if (action == BUS_NOTIFY_DEL_DEVICE) {
-		if (ops->remove_device && dev->iommu_group) {
+		if (ops->remove_device) {
  			ops->remove_device(dev);
  			return 0;
  		}
include/linux/iommu.h +2 -2
index f3b99e1..2715592 100644
@@ -74,8 +74,8 @@ enum iommu_attr {
   * @unmap: unmap a physically contiguous memory region from an iommu domain
   * @iova_to_phys: translate iova to physical address
   * @domain_has_cap: domain capabilities query
- * @add_device: add device to iommu grouping
- * @remove_device: remove device from iommu grouping
+ * @add_device: add device to iommu grouping or domain
+ * @remove_device: remove device from iommu grouping or domain
   * @domain_get_attr: Query domain attributes
   * @domain_set_attr: Change domain attributes
   * @pgsize_bitmap: bitmap of supported page sizes

A message that is not in the archive (sent off-list or not yet received): 1358457711.9252.54.camel-85EaTFmN5p//[email protected]

#9 lore raw

Hi Alex,

Quoted 36 lines · Alex Williamson
Quoted 27 lines · Damian Hobson-Garcia

I mistakenly sent this with the tag [PATCH] when it should have been [RFC]. Updated with this reply.

Hi Alex,

Quoted 22 lines · Alex Williamson
Quoted 14 lines · Damian Hobson-Garcia

We've recently been thinking about the best way to attach platform devices in
an IOMMU in a generic way without having to have every platform device
register itself at probe time. We've seen that at least on the tegra IOMMU,
using the platform_bus notifier seems like a viable option (http://lists.linuxfoundation.org/pipermail/iommu/2012-December/004954.html). We also noticed that the IOMMU ops add_device() function is called (indirectly) from a similar bus
notifier and we're thinking that using the bus notifier in the common code
would be better than us rolling our own to do basically the same thing.

The problem, however, is that add_device() seems to be intended for use with IOMMU groups. i.e. After calling add_device() on a device that is should be in a group, dev->iommu_group should have some meaningful value.
I think though, that if the IOMMU groups requirement can be removed, it should be more versatile.

Quoted 3 lines

I believe that removing the IOMMU groups dependency can be achieved with the patch below. Currently the intel-iommu seems to be the only implementation using these callbacks.

What stops you from creating a single iommu_group and adding all device to it? Thanks,

Quoted 7 lines
Quoted 1 line

Alex

Thanks for your comment.

Nothing stops me, but putting everything in one group doesn't really add any benefit over not using groups at all, does it? It seems like its just adding more complexity(i.e code) to the driver without adding any additional functionality.

Quoted 8 lines

Does your iommu provide any isolation or is it translate-only. As Varun notes, groups enable vfio, so if there's any possibility that you care about userspace drivers for your devices or assigning devices to qemu guests, then groups might be useful. The overhead of groups is really meant to be very minimal for the iommu driver. That said, I guess I have no real objection to generalizing the iommu bus notifier for add/remove. Thanks,

Our iommu is used for translation. We plan on using vfio at some point, though how well that fits with the current IOMMU ARM implementation remains to be seen. So considering your and Varun's comments, it looks like we want to be implementing groups after all, though generalizing the add/remove still seems to me like a good idea.

Thanks,
Damian

Quoted 59 lines

Alex

Quoted 54 lines
Quoted 52 lines
Quoted 49 lines
---
   drivers/iommu/intel-iommu.c |    3 ++-
   drivers/iommu/iommu.c       |    2 +-
   include/linux/iommu.h       |    4 ++--
   3 files changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +2 -1
index b9d0911..a763872 100644
@@ -4217,7 +4217,8 @@ root_bus:
 
   static void intel_iommu_remove_device(struct device *dev)
   {
-	iommu_group_remove_device(dev);
+	if (dev->iommu_group)
+		iommu_group_remove_device(dev);
   }
 
   static struct iommu_ops intel_iommu_ops = {
drivers/iommu/iommu.c +1 -1
index ddbdaca..a6046c2 100644
@@ -510,7 +510,7 @@ static int iommu_bus_notifier(struct notifier_block *nb,
   		if (ops->add_device)
   			return ops->add_device(dev);
   	} else if (action == BUS_NOTIFY_DEL_DEVICE) {
-		if (ops->remove_device && dev->iommu_group) {
+		if (ops->remove_device) {
   			ops->remove_device(dev);
   			return 0;
   		}
include/linux/iommu.h +2 -2
index f3b99e1..2715592 100644
@@ -74,8 +74,8 @@ enum iommu_attr {
    * @unmap: unmap a physically contiguous memory region from an iommu domain
    * @iova_to_phys: translate iova to physical address
    * @domain_has_cap: domain capabilities query
- * @add_device: add device to iommu grouping
- * @remove_device: remove device from iommu grouping
+ * @add_device: add device to iommu grouping or domain
+ * @remove_device: remove device from iommu grouping or domain
    * @domain_get_attr: Query domain attributes
    * @domain_set_attr: Change domain attributes
    * @pgsize_bitmap: bitmap of supported page sizes
Signature · 3 lines
Damian Hobson-Garcia
IGEL Co.,Ltd
http://www.igel.co.jp