vfio/mdev: IOMMU aware mediated device

Add the response to IOMMU_CAP_AUX_DOMAIN capability query through iommu_capable(). Return true if IOMMUs support the scalable mode, return false otherwise.

Cc Ashok Raj <[email protected]>

Cc Jacob Pan <[email protected]>

Cc Kevin Tian <[email protected]>

Cc Liu Yi L <[email protected]>

Signed-off-by Lu Baolu <[email protected]>

---
 drivers/iommu/intel-iommu.c | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

drivers/iommu/intel-iommu.c +20 -1

index 3e49d4029058..891ae70e7bf2 100644
@@ -5193,12 +5193,39 @@ static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain,
 	return phys;
 }
 
+static inline bool scalable_mode_support(void)
+{
+	struct dmar_drhd_unit *drhd;
+	struct intel_iommu *iommu;
+	bool ret = true;
+
+	rcu_read_lock();
+	for_each_active_iommu(iommu, drhd) {
+		if (!sm_supported(iommu)) {
+			ret = false;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
+	return ret;
+}
+
 static bool intel_iommu_capable(enum iommu_cap cap)
 {
-	if (cap == IOMMU_CAP_CACHE_COHERENCY)
+	switch (cap) {
+	case IOMMU_CAP_CACHE_COHERENCY:
 		return domain_update_iommu_snooping(NULL) == 1;

Quoted 52 lines · On Thu, 30 Aug 2018 12:09:14 +0800 Lu Baolu…

Add the response to IOMMU_CAP_AUX_DOMAIN capability query through iommu_capable(). Return true if IOMMUs support the scalable mode, return false otherwise.

Cc Ashok Raj <[email protected]>

Cc Jacob Pan <jacob.jun.pan-VuQAYsv1563Yd54FQh9/[email protected]>

Cc Kevin Tian <[email protected]>

Cc Liu Yi L <[email protected]>

Signed-off-by Lu Baolu <baolu.lu-VuQAYsv1563Yd54FQh9/[email protected]>

---
  drivers/iommu/intel-iommu.c | 31 +++++++++++++++++++++++++++++--
  1 file changed, 29 insertions(+), 2 deletions(-)

drivers/iommu/intel-iommu.c +25 -2

index 3e49d4029058..891ae70e7bf2 100644
@@ -5193,12 +5193,39 @@ static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain,
  	return phys;
  }
 
+static inline bool scalable_mode_support(void)
+{
+	struct dmar_drhd_unit *drhd;
+	struct intel_iommu *iommu;
+	bool ret = true;
+
+	rcu_read_lock();
+	for_each_active_iommu(iommu, drhd) {
+		if (!sm_supported(iommu)) {
+			ret = false;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
+	return ret;
+}
+
  static bool intel_iommu_capable(enum iommu_cap cap)
  {
-	if (cap == IOMMU_CAP_CACHE_COHERENCY)
+	switch (cap) {
+	case IOMMU_CAP_CACHE_COHERENCY:
  		return domain_update_iommu_snooping(NULL) == 1;
-	if (cap == IOMMU_CAP_INTR_REMAP)
+	case IOMMU_CAP_INTR_REMAP:
  		return irq_remapping_enabled == 1;
+	case IOMMU_CAP_AUX_DOMAIN:
+		return scalable_mode_support();
+	case IOMMU_CAP_NOEXEC:
+	/* PASSTHROUGH */

drivers/iommu/intel-iommu.c static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain, · line 5227

Quoted 3 lines

drivers/iommu/intel-iommu.c +3 -0

+	default:
+		pr_info("Unsupported capability query %d\n", cap);
+		break;

Please don't do this, there's no reason to be noisy about a query of a capability that VT-d doesn't know about. We implement capabilities exactly so that relevant drivers can expose a feature and others can happily (and quietly) ignore them. Thanks,

Otherwise, there will be a build warning:

drivers/iommu/amd_iommu.c:3083:2: warning: enumeration value 'IOMMU_CAP_AUX_DOMAIN' not handled in switch [-Wswitch]

There is no functional change.

Signed-off-by Lu Baolu <baolu.lu-VuQAYsv1563Yd54FQh9/[email protected]>

---
 drivers/iommu/amd_iommu.c | 2 ++
 1 file changed, 2 insertions(+)

drivers/iommu/amd_iommu.c +0 -0

index 4e04fff23977..237ae6db4cfd 100644

Quoted 24 lines · On Thu, 30 Aug 2018 12:09:15 +0800 Lu Baolu…

Otherwise, there will be a build warning:

drivers/iommu/amd_iommu.c:3083:2: warning: enumeration value 'IOMMU_CAP_AUX_DOMAIN' not handled in switch [-Wswitch]

There is no functional change.

Signed-off-by Lu Baolu <baolu.lu-VuQAYsv1563Yd54FQh9/[email protected]>

---
  drivers/iommu/amd_iommu.c | 2 ++
  1 file changed, 2 insertions(+)

drivers/iommu/amd_iommu.c +2 -0

index 4e04fff23977..237ae6db4cfd 100644
@@ -3077,6 +3077,8 @@ static bool amd_iommu_capable(enum iommu_cap cap)
  		return (irq_remapping_enabled == 1);
  	case IOMMU_CAP_NOEXEC:
  		return false;
+	default:
+		break;
  	}
 

drivers/iommu/amd_iommu.c static bool amd_iommu_capable(enum iommu_cap cap) · line 3084

Quoted 1 line

drivers/iommu/amd_iommu.c +0 -0

  	return false;

Seems like a bug fix that doesn't need to be part of this RFC, send it separately. Thanks,

drivers/vfio/vfio_iommu_type1.c static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base) · line 1367

Quoted 6 lines · Lu Baolu

drivers/vfio/vfio_iommu_type1.c +6 -0

+static int vfio_detach_aux_domain(struct device *dev, void *data)
+{
+	struct iommu_domain *domain = data;
+
+	vfio_mdev_set_aux_domain(dev, NULL);
+	iommu_detach_device(domain, dev->parent);

I think that's only going to work for vt-d, which doesn't use a default_domain. For other drivers, iommu.c ends up calling domain->ops->attach_dev(default_domain, dev) here, which isn't what we want.

That was my concern with reusing attach/detach_dev callbacks for PASID management. The attach_dev code of IOMMU drivers already has to deal with toggling between default and unmanaged domain. Dealing with more state transitions in the same path is going to be difficult.

Hi,

Quoted 11 lines · Jean-Philippe Brucker

drivers/vfio/vfio_iommu_type1.c static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base) · line 1367

Quoted 6 lines · Lu Baolu

drivers/vfio/vfio_iommu_type1.c +6 -0

+static int vfio_detach_aux_domain(struct device *dev, void *data)
+{
+	struct iommu_domain *domain = data;
+
+	vfio_mdev_set_aux_domain(dev, NULL);
+	iommu_detach_device(domain, dev->parent);

I think that's only going to work for vt-d, which doesn't use a default_domain. For other drivers, iommu.c ends up calling domain->ops->attach_dev(default_domain, dev) here, which isn't what we want.

This doesn't break any functionality. It takes effect only if iommu hardware supports finer granular translation and advertises it through the API.>

Quoted 4 lines

That was my concern with reusing attach/detach_dev callbacks for PASID management. The attach_dev code of IOMMU drivers already has to deal with toggling between default and unmanaged domain. Dealing with more state transitions in the same path is going to be difficult.

Let's consider it from the API point of view.

We have iommu_a(de)ttach_device() APIs to attach or detach a domain to/from a device. We should avoid applying a limitation of "these are only for single domain case, for multiple domains, use another API".

From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Thursday, September 13, 2018 1:54 AM

Quoted 14 lines · Lu Baolu

Hi,

Quoted 11 lines · Jean-Philippe Brucker

drivers/vfio/vfio_iommu_type1.c static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base) · line 1367

Quoted 6 lines · Lu Baolu

drivers/vfio/vfio_iommu_type1.c +6 -0

+static int vfio_detach_aux_domain(struct device *dev, void *data)
+{
+	struct iommu_domain *domain = data;
+
+	vfio_mdev_set_aux_domain(dev, NULL);
+	iommu_detach_device(domain, dev->parent);

I think that's only going to work for vt-d, which doesn't use a default_domain. For other drivers, iommu.c ends up calling domain->ops->attach_dev(default_domain, dev) here, which isn't what

we want.

Quoted 5 lines

This doesn't break any functionality. It takes effect only if iommu hardware supports finer granular translation and advertises it through the API.>

Quoted 1 line

That was my concern with reusing attach/detach_dev callbacks for

PASID

Quoted 1 line

Quoted 1 line

management. The attach_dev code of IOMMU drivers already has to

deal

Quoted 1 line

Quoted 1 line

with toggling between default and unmanaged domain. Dealing with

more

Quoted 7 lines

Quoted 1 line

state transitions in the same path is going to be difficult.

Let's consider it from the API point of view.

We have iommu_a(de)ttach_device() APIs to attach or detach a domain to/from a device. We should avoid applying a limitation of "these are only for single domain case, for multiple domains, use another API".

That's an idealized view of the API, the actual semantics aren't as simple. For IOMMU drivers that implement IOMMU_DOMAIN_DMA in their domain_alloc operation (Arm SMMU, AMD IOMMU, ...), attach_dev attaches
an unmanaged domain, detach_dev reattaches the default DMA domain, and
the detach_dev IOMMU op is not called. We can't change that now, so it's difficult to add more functionality to attach_dev and detach_dev.

Quoted 6 lines

Quoted 5 lines

Let's consider it from the API point of view.

We have iommu_a(de)ttach_device() APIs to attach or detach a domain to/from a device. We should avoid applying a limitation of "these are only for single domain case, for multiple domains, use another API".

Quoted 9 lines

That's an idealized view of the API, the actual semantics aren't as simple. For IOMMU drivers that implement IOMMU_DOMAIN_DMA in their domain_alloc operation (Arm SMMU, AMD IOMMU, ...), attach_dev attaches
an unmanaged domain, detach_dev reattaches the default DMA domain, and
the detach_dev IOMMU op is not called. We can't change that now, so it's difficult to add more functionality to attach_dev and detach_dev.

Now we have four possible usages on a(de)ttach_device:

1) Normal DMA API path i.e. IOMMU_DOMAIN_DMA, for DMA operations in host/parent device driver;

2) VFIO passthru path, when the physical device is assigned to user space or guest driver

3) mdev passthru path 1, when mdev is assigned to user space or guest driver. Here mdev is a wrap on random PCI device

4) mdev passthru path 2, when mdev is assigned to user space or guest driver. Here mdev is in a smaller granularity (e.g. tagged by PASID) as supported by vt-d scalable mode

1) and 2) are existing usages. What you described (unmanaged vs. default) falls into this category.

3) is actually same as 2) in IOMMU layer. sort of passing through DMA capability to guest. Though there is still a parent driver, the parent driver itself should not do DMA - i.e. unmanaged in host side.

4) is a new code path introduced in IOMMU layer, which is what vfio_a(de)tach_aux_domain is trying to address. In this case parent device can still have its own DMA capability, using existing code path 1) (thus default domain still applies). and the parent device can have multiple aux domains (and associated structures), using code path 4).

From: Jean-Philippe Brucker
Sent: Friday, September 14, 2018 10:46 PM

Quoted 8 lines · Kevin

Quoted 8 lines

Quoted 5 lines

Let's consider it from the API point of view.

We have iommu_a(de)ttach_device() APIs to attach or detach a domain to/from a device. We should avoid applying a limitation of "these are only for single domain case, for multiple domains, use another API".

That's an idealized view of the API, the actual semantics aren't as simple. For IOMMU drivers that implement IOMMU_DOMAIN_DMA in

their

Quoted 5 lines

Quoted 5 lines

domain_alloc operation (Arm SMMU, AMD IOMMU, ...), attach_dev attaches
an unmanaged domain, detach_dev reattaches the default DMA domain, and
the detach_dev IOMMU op is not called. We can't change that now, so

it's

Quoted 6 lines

Quoted 2 lines

difficult to add more functionality to attach_dev and detach_dev.

Now we have four possible usages on a(de)ttach_device:

1) Normal DMA API path i.e. IOMMU_DOMAIN_DMA, for DMA

operations

Quoted 15 lines

in host/parent device driver;

2) VFIO passthru path, when the physical device is assigned to user space or guest driver

3) mdev passthru path 1, when mdev is assigned to user space or guest driver. Here mdev is a wrap on random PCI device

4) mdev passthru path 2, when mdev is assigned to user space or guest driver. Here mdev is in a smaller granularity (e.g. tagged by PASID) as supported by vt-d scalable mode

1) and 2) are existing usages. What you described (unmanaged vs. default) falls into this category.

Quoted 6 lines

3) is actually same as 2) in IOMMU layer. sort of passing through DMA capability to guest. Though there is still a parent driver, the parent driver itself should not do DMA - i.e. unmanaged in host side.

4) is a new code path introduced in IOMMU layer, which is what vfio_a(de)tach_aux_domain is trying to address. In this case parent

device

Quoted 3 lines

can still have its own DMA capability, using existing code path 1) (thus default domain still applies). and the parent device can have multiple aux domains (and associated structures), using code path 4).

We still have the problem that detach() doesn't propagate to the IOMMU driver. Here's the current flow, without mdev:

1) At boot, the PF's (parent device) driver is probed, and the IOMMU core sets up a default DMA domain, to be used by dma_alloc by the PF's driver.
-> iommu.c calls default_domain->ops->attach_dev(default_domain, dev)

2) or 3) Later userspace wants to control the PF, replaces the PF's driver with vfio-pci. When userspace creates a container, VFIO allocates a new IOMMU domain, and calls iommu_attach_group.
-> iommu.c calls domain->ops->attach_dev(domain, dev)
This detaches the PF from the default domain, and attaches it to the new domain.

1) When the container is closed, VFIO calls iommu_detach_group. This detaches the PF from its current domain, and attaches it back to the default domain.
-> iommu.c calls default_domain->ops->attach_dev(default_domain, dev)

-----
Now with mdev, we still attach the DMA domain in 1). Then:

4) Userspace opens an mdev and creates a container. VFIO enables aux domain for the device. VFIO allocates a new IOMMU domain, and calls iommu_attach_device(domain1, parent_dev).
-> iommu.c calls domain->ops->attach_dev(domain1, dev) Because the device is in "aux domain" state, the IOMMU driver does not detach from the default domain, but instead allocates a PASID and attaches the aux domain. (Side note: for SMMU we couldn't detach from the default domain, because we need it for MSI mappings.)

4) Userspace opens another mdev.
-> iommu.c calls domain->ops->attach_dev(domain2, dev)

1)? When the container is closed, VFIO calls iommu_detach_device(domain2, parent_dev)
-> iommu.c calls default_domain->ops->attach_dev(default_domain, dev) Given that auxiliary domains are attached, the IOMMU driver could deduce that this actually means "detach an auxiliary domain". But which one?

So the proposed interface doesn't seem to work as is. If we want to use iommu_attach/detach_device for auxiliary domains, the existing behavior of iommu.c, and IOMMU drivers that rely on it, have to change. Any change I can think of right now seems more daunting than introducing a different path for auxiliary domains, like iommu_attach_aux_domain for example.

Quoted 2 lines

4) Userspace opens another mdev.
-> iommu.c calls domain->ops->attach_dev(domain2, dev)

another mdev in same VFIO container or different? I assume the latter since you mentioned a new domain2.

Quoted 5 lines

1)? When the container is closed, VFIO calls iommu_detach_device(domain2, parent_dev)
-> iommu.c calls default_domain->ops->attach_dev(default_domain, dev) Given that auxiliary domains are attached, the IOMMU driver could deduce that this actually means "detach an auxiliary domain". But which one?

I didn't get this one. There is no need to stick to 1) behavior for 4), i.e. below is expected:
        domain2->ops->detach_dev(domain2, dev)

why cannot ARM implement a detach_dev for aux_domain too? My feeling is that default domain twist is only for switch between 1/2/3 in concept.

Quoted 7 lines

So the proposed interface doesn't seem to work as is. If we want to use iommu_attach/detach_device for auxiliary domains, the existing behavior of iommu.c, and IOMMU drivers that rely on it, have to change. Any change I can think of right now seems more daunting than introducing a different path for auxiliary domains, like iommu_attach_aux_domain for example.

introducing *aux* specific API will cause different VFIO code path to handle RID-based and PASID-based mdev, since RID-based still needs to use normal attach_domain that way.

well, this argument is not very strong
in itself, if indeed proposed way doesn't work for ARM. But let's see whether it is the case with more understanding of your actual concern.

Thanks
Kevin

From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Tuesday, September 18, 2018 11:52 PM

Quoted 5 lines · Kevin

Quoted 2 lines

4) Userspace opens another mdev.
-> iommu.c calls domain->ops->attach_dev(domain2, dev)

another mdev in same VFIO container or different? I assume the latter since you mentioned a new domain2.

I was thinking a different VFIO container actually. I used domain2 to try to make the example clearer

Quoted 3 lines

Quoted 3 lines

1)? When the container is closed, VFIO calls iommu_detach_device(domain2, parent_dev)
-> iommu.c calls default_domain->ops->attach_dev(default_domain,

dev)

Quoted 1 line

Quoted 1 line

Given that auxiliary domains are attached, the IOMMU driver could

deduce

Quoted 5 lines

Quoted 1 line

that this actually means "detach an auxiliary domain". But which one?

I didn't get this one. There is no need to stick to 1) behavior for 4), i.e. below is expected:
        domain2->ops->detach_dev(domain2, dev)

But in order to get that, the IOMMU core needs to know that domain2 is auxiliary. Otherwise, detach_dev is never called when a default_domain is present for the parent dev.

I guess one solution is to add an "auxiliary" attribute to iommu_domain, so __iommu_detach_group would do something like:

drivers/iommu/iommu.c +3 -2

index 7113fe398b70..2b3e9b91aee7 100644
@@ -1786,10 +1786,11 @@ static void __iommu_detach_group(struct
 iommu_domain *domain,
 {
 	int ret;
 
-	if (!group->default_domain) {
+	if (!group->default_domain || domain->auxiliary) {
 		__iommu_group_for_each_dev(group, domain,
 					   iommu_group_do_detach_device);
-		group->domain = NULL;
+		if (!domain->auxiliary)
+			group->domain = NULL;
 		return;
 	}

Not sure who would set this "auxiliary" attribute... Maybe the IOMMU driver, when attaching the domain to a device that has auxiliary mode enabled?

Quoted 3 lines

why cannot ARM implement a detach_dev for aux_domain too? My feeling is that default domain twist is only for switch between 1/2/3 in concept.

If the core actually calls it, we can implement detach_dev :) The problem is that the core never calls detach_dev when default_domain is present (affects any IOMMU driver that relies on default_domain, including AMD), and even in case 4) the default_domain is present for the parent device

Quoted 3 lines

Quoted 3 lines

So the proposed interface doesn't seem to work as is. If we want to use iommu_attach/detach_device for auxiliary domains, the existing

behavior

Quoted 6 lines

Quoted 5 lines

of iommu.c, and IOMMU drivers that rely on it, have to change. Any change I can think of right now seems more daunting than introducing a different path for auxiliary domains, like iommu_attach_aux_domain for example.

Quoted 3 lines

introducing *aux* specific API will cause different VFIO code path to handle RID-based and PASID-based mdev, since RID-based still needs to use normal attach_domain that way.

The PASID-based mdev still requires a special case to retrieve the allocated PASID and program it in the parent device, so VFIO will need to know the difference between the two

Quoted 29 lines

From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Tuesday, September 18, 2018 11:52 PM

Quoted 5 lines · Kevin

Quoted 2 lines

4) Userspace opens another mdev.
-> iommu.c calls domain->ops->attach_dev(domain2, dev)

another mdev in same VFIO container or different? I assume the latter since you mentioned a new domain2.

I was thinking a different VFIO container actually. I used domain2 to try to make the example clearer

Quoted 3 lines

Quoted 3 lines

1)? When the container is closed, VFIO calls iommu_detach_device(domain2, parent_dev)
-> iommu.c calls default_domain->ops->attach_dev(default_domain,

dev)

Quoted 1 line

Quoted 1 line

Given that auxiliary domains are attached, the IOMMU driver could

deduce

Quoted 5 lines

Quoted 1 line

that this actually means "detach an auxiliary domain". But which one?

I didn't get this one. There is no need to stick to 1) behavior for 4), i.e. below is expected:
        domain2->ops->detach_dev(domain2, dev)

But in order to get that, the IOMMU core needs to know that domain2 is auxiliary. Otherwise, detach_dev is never called when a default_domain is present for the parent dev.

Quoted 2 lines

I guess one solution is to add an "auxiliary" attribute to iommu_domain, so __iommu_detach_group would do something like:

this doesn't work. same domain can be also attached to another physical device as non-aux domain (e.g. passthrough) at the same time (vfio-pci device and vfio-mdev device in same container), then default domain tweak is required in that case. "aux" takes effect only per-device, not per-domain.

Quoted 29 lines

drivers/iommu/iommu.c +3 -2

index 7113fe398b70..2b3e9b91aee7 100644
@@ -1786,10 +1786,11 @@ static void __iommu_detach_group(struct
 iommu_domain *domain,
  {
  	int ret;
 
-	if (!group->default_domain) {
+	if (!group->default_domain || domain->auxiliary) {
  		__iommu_group_for_each_dev(group, domain,
  					   iommu_group_do_detach_device);
-		group->domain = NULL;
+		if (!domain->auxiliary)
+			group->domain = NULL;
  		return;
  	}

Not sure who would set this "auxiliary" attribute... Maybe the IOMMU driver, when attaching the domain to a device that has auxiliary mode enabled?

Quoted 3 lines

why cannot ARM implement a detach_dev for aux_domain too? My feeling is that default domain twist is only for switch between 1/2/3 in concept.

Quoted 5 lines

If the core actually calls it, we can implement detach_dev :) The problem is that the core never calls detach_dev when default_domain is present (affects any IOMMU driver that relies on default_domain, including AMD), and even in case 4) the default_domain is present for the parent device

Then can we change that core logic so detach_dev is invoked in all cases? yes there will be some changes in vendor drivers, but I expect this change trivial (especially considering the gain in IOMMU API simplicity side as described below).

Quoted 10 lines

Quoted 3 lines

Quoted 3 lines

So the proposed interface doesn't seem to work as is. If we want to use iommu_attach/detach_device for auxiliary domains, the existing

behavior

Quoted 6 lines

Quoted 5 lines

of iommu.c, and IOMMU drivers that rely on it, have to change. Any change I can think of right now seems more daunting than introducing a different path for auxiliary domains, like iommu_attach_aux_domain for example.

Quoted 8 lines

Quoted 3 lines

introducing *aux* specific API will cause different VFIO code path to handle RID-based and PASID-based mdev, since RID-based still needs to use normal attach_domain that way.

The PASID-based mdev still requires a special case to retrieve the allocated PASID and program it in the parent device, so VFIO will need to know the difference between the two

that retrieve/program is down by parent driver, instead of VFIO.

Thanks
Kevin

Quoted 5 lines

If the core actually calls it, we can implement detach_dev :) The problem is that the core never calls detach_dev when default_domain is present (affects any IOMMU driver that relies on default_domain, including AMD), and even in case 4) the default_domain is present for the parent device

Then can we change that core logic so detach_dev is invoked in all cases? yes there will be some changes in vendor drivers, but I expect this change trivial (especially considering the gain in IOMMU API simplicity side as described below).

Quoted 10 lines · Kevin

Quoted 5 lines

If the core actually calls it, we can implement detach_dev :) The problem is that the core never calls detach_dev when default_domain is present (affects any IOMMU driver that relies on default_domain, including AMD), and even in case 4) the default_domain is present for the parent device

Then can we change that core logic so detach_dev is invoked in all cases? yes there will be some changes in vendor drivers, but I expect this change trivial (especially considering the gain in IOMMU API simplicity side as described below).

Thinking more about this, there might be a way that doesn't require too much rewriting of IOMMU drivers. When VFIO calls iommu_detach_group to detach a vfio-pci device from its domain, it could do detach_dev(dev, old_domain) followed by attach_dev(dev, default_domain) instead of only attach_dev(dev, default_domain), so:

__iommu_attach_group(grp, unmanaged_domain)
ops->attach_dev(dev, unmanaged_domain)
-> IOMMU driver detaches from default_domain only if the dev isn't
in auxiliary mode

__iommu_detach_group(grp, unmanaged_domain)
if (ops->detach_dev)

       ops->detach_dev(dev, unmanaged_domain)
   ops->attach_dev(dev, default_domain)
     -> IOMMU driver ignores this if still attached to default_domain

It's not trivial: since half the IOMMU drivers seem to be using default domain now, their detach_dev callback might have been dead code for a while (I can't find a path where it still gets called), so re-enabling it requires some attention. However, since most of them won't care about aux domains, maybe we could just remove their detach_dev pointer.

The resulting logic of attach/detach_dev in IOMMU drivers that do use aux domains still seems unnecessarily convoluted. It could as well be done with separate attach_aux/detach_aux functions. If you want to keep it that way though, I think the above could work for us and I'll try to write an RFC.

Thanks,
Jean

From: Lu Baolu [mailto:baolu.lu-VuQAYsv1563Yd54FQh9/[email protected]]
Sent: Thursday, August 30, 2018 12:09 PM

In order to distinguish the IOMMU-capable mediated devices from those which still need to rely on parent devices, this patch set adds a domain type attribute to each mdev.

enum mdev_domain_type {
DOMAIN_TYPE_NO_IOMMU, /* Don't need any IOMMU support.
* All isolation and protection
* are handled by the parent
* device driver with a device
* specific mechanism. */
DOMAIN_TYPE_ATTACH_PARENT, /* IOMMU can isolate and
protect
* the mdev, and the isolation
* domain should be attaced with
* the parent device. */
};

Quoted 3 lines

From: Lu Baolu [mailto:[email protected]]
Sent: Thursday, August 30, 2018 12:09 PM

[...]

Quoted 5 lines

In order to distinguish the IOMMU-capable mediated devices from those which still need to rely on parent devices, this patch set adds a domain type attribute to each mdev.

Quoted 15 lines

enum mdev_domain_type {
DOMAIN_TYPE_NO_IOMMU, /* Don't need any IOMMU support.
* All isolation and protection
* are handled by the parent
* device driver with a device
* specific mechanism. */
DOMAIN_TYPE_ATTACH_PARENT, /* IOMMU can isolate and
protect
* the mdev, and the isolation
* domain should be attaced with
* the parent device. */
};

ATTACH_PARENT is not like a good counterpart to NO_IOMMU.

what about DOMAIN_TYPE_NO_IOMMU/DOMAIN_TYPE_IOMMU? whether to attach parent device is just internal logic.

Alternatively DOMAIN_TYPE_SOFTWARE/DOMAIN_TYPE_HARDWARE, where software means iommu_domain is managed by software while the other means managed by hardware.

One side note to Alex - with multiple domain extension in IOMMU layer, this version combines IOMMU-capable usages in VFIO: PASID-based (as in scalable iov) and RID-based (as the usage of mdev wrapper on any device). Both cases share the common path - just binding the domain to the parent device of mdev. IOMMU layer will handle two cases differently later.

Quoted 25 lines · Kevin

Quoted 3 lines

From: Lu Baolu [mailto:[email protected]]
Sent: Thursday, August 30, 2018 12:09 PM

[...]

Quoted 20 lines

In order to distinguish the IOMMU-capable mediated devices from those which still need to rely on parent devices, this patch set adds a domain type attribute to each mdev.

enum mdev_domain_type {
DOMAIN_TYPE_NO_IOMMU, /* Don't need any IOMMU support.
* All isolation and protection
* are handled by the parent
* device driver with a device
* specific mechanism. */
DOMAIN_TYPE_ATTACH_PARENT, /* IOMMU can isolate and
protect
* the mdev, and the isolation
* domain should be attaced with
* the parent device. */
};

Quoted 1 line

ATTACH_PARENT is not like a good counterpart to NO_IOMMU.

Please do not use NO_IOMMU, we already have a thing called vfio-noiommu, enabled through CONFIG_VFIO_NOIOMMU and module parameter enable_unsafe_noiommu_mode. This is much, much too similar and will generate confusion.

Quoted 7 lines

what about DOMAIN_TYPE_NO_IOMMU/DOMAIN_TYPE_IOMMU? whether to attach parent device is just internal logic.

Alternatively DOMAIN_TYPE_SOFTWARE/DOMAIN_TYPE_HARDWARE, where software means iommu_domain is managed by software while the other means managed by hardware.

I haven't gotten deep enough into the series to see how it's used, but my gut reaction is that we don't need an enum, we just need some sort of pointer on the mdev that points to an iommu_parent, which indicates the root of our IOMMU based isolation, or is NULL, which indicates we use vendor defined isolation as we have now.

Quoted 6 lines

One side note to Alex - with multiple domain extension in IOMMU layer, this version combines IOMMU-capable usages in VFIO: PASID-based (as in scalable iov) and RID-based (as the usage of mdev wrapper on any device). Both cases share the common path - just binding the domain to the parent device of mdev. IOMMU layer will handle two cases differently later.

Good, I'm glad you've considered the regular (RID) IOMMU domain and not just the new aux domain. Thanks,

Alex

Below APIs are introduced in the IOMMU glue for device drivers to use the finer granularity translation.

* iommu_capable(IOMMU_CAP_AUX_DOMAIN)
- Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware.

* iommu_en(dis)able_aux_domain(struct device *dev)
- Enable/disable the multiple domains capability for a device referenced by @dev.

* iommu_auxiliary_id(struct iommu_domain *domain)
- Return the index value used for finer-granularity DMA translation. The specific device driver needs to feed the hardware with this value, so that hardware device could issue the DMA transaction with this value tagged.

Hi,

Quoted 3 lines · Lu Baolu

Below APIs are introduced in the IOMMU glue for device drivers to use the finer granularity translation.

Quoted 3 lines

* iommu_capable(IOMMU_CAP_AUX_DOMAIN)
- Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware.

iommu_capable() cannot represent hardware capabilities, we need something else for systems with multiple IOMMUs that have different caps. How about iommu_domain_get_attr on the device's domain instead?

Quoted 10 lines

* iommu_en(dis)able_aux_domain(struct device *dev)
- Enable/disable the multiple domains capability for a device referenced by @dev.

* iommu_auxiliary_id(struct iommu_domain *domain)
- Return the index value used for finer-granularity DMA translation. The specific device driver needs to feed the hardware with this value, so that hardware device could issue the DMA transaction with this value tagged.

This could also reuse iommu_domain_get_attr.

More generally I'm having trouble understanding how auxiliary domains will be used. So VFIO allocates PASIDs like this:

* iommu_enable_aux_domain(parent_dev)
* iommu_domain_alloc() -> dom1
* iommu_domain_alloc() -> dom2
* iommu_attach_device(dom1, parent_dev)
-> dom1 gets PASID #1
* iommu_attach_device(dom2, parent_dev)
-> dom2 gets PASID #2

Then I'm not sure about the next steps, when userspace does VFIO_IOMMU_MAP_DMA or VFIO_IOMMU_BIND on an mdev's container. Is the following use accurate?

For the single translation level:
* iommu_map(dom1, ...) updates first-level/second-level pgtables for PASID #1
* iommu_map(dom2, ...) updates first-level/second-level pgtables for PASID #2

Nested translation:
* iommu_map(dom1, ...) updates second-level pgtables for PASID #1
* iommu_bind_table(dom1, ...) binds first-level pgtables, provided by the guest, for PASID #1
* iommu_map(dom2, ...) updates second-level pgtables for PASID #2
* iommu_bind_table(dom2, ...) binds first-level pgtables for PASID #2

Quoted 1 line

I'm trying to understand how to implement this with SMMU and other

IOMMUs. It's not a clean fit since we have a single domain to hold the second-level pgtables.

Then again, the nested case probably doesn't
matter for us - we might as well assign the parent directly, since all mdevs have the same second-level and can only be assigned to the same VM.

Also, can non-VFIO device drivers use auxiliary domains to do map/unmap on PASIDs? They are asking to do that and I'm proposing the private PASID thing, but since aux domains provide a similar feature we should probably converge somehow.

Hi,

Quoted 6 lines · Jean-Philippe Brucker

Hi,

Quoted 3 lines · Lu Baolu

Below APIs are introduced in the IOMMU glue for device drivers to use the finer granularity translation.

Quoted 7 lines

Quoted 3 lines

* iommu_capable(IOMMU_CAP_AUX_DOMAIN)
- Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware.

iommu_capable() cannot represent hardware capabilities, we need something else for systems with multiple IOMMUs that have different caps. How about iommu_domain_get_attr on the device's domain instead?

Domain is not a good choice for per iommu cap query. A domain might be attached to devices belonging to different iommu's.

How about an API with device structure as parameter? A device always belongs to a specific iommu. This API is supposed to be used the device driver.

Quoted 3 lines

Quoted 3 lines

* iommu_en(dis)able_aux_domain(struct device *dev)
- Enable/disable the multiple domains capability for a device referenced by @dev.

Quoted 11 lines

Quoted 5 lines

* iommu_auxiliary_id(struct iommu_domain *domain)
- Return the index value used for finer-granularity DMA translation. The specific device driver needs to feed the hardware with this value, so that hardware device could issue the DMA transaction with this value tagged.

This could also reuse iommu_domain_get_attr.

More generally I'm having trouble understanding how auxiliary domains will be used. So VFIO allocates PASIDs like this:

As I wrote in the cover letter, "auxiliary domain" is just a name to ease discussion. It's actually has no special meaning (we think a domain as an isolation boundary which could be used by the IOMMU to isolate the DMA transactions out of a PCI device or partial of it).

So drivers like vfio should see no difference when use an auxiliary domain. The auxiliary domain is not aware out of iommu driver.

Quoted 25 lines

* iommu_enable_aux_domain(parent_dev)
* iommu_domain_alloc() -> dom1
* iommu_domain_alloc() -> dom2
* iommu_attach_device(dom1, parent_dev)
-> dom1 gets PASID #1
* iommu_attach_device(dom2, parent_dev)
-> dom2 gets PASID #2

Then I'm not sure about the next steps, when userspace does VFIO_IOMMU_MAP_DMA or VFIO_IOMMU_BIND on an mdev's container. Is the following use accurate?

For the single translation level:
* iommu_map(dom1, ...) updates first-level/second-level pgtables for PASID #1
* iommu_map(dom2, ...) updates first-level/second-level pgtables for PASID #2

Nested translation:
* iommu_map(dom1, ...) updates second-level pgtables for PASID #1
* iommu_bind_table(dom1, ...) binds first-level pgtables, provided by the guest, for PASID #1
* iommu_map(dom2, ...) updates second-level pgtables for PASID #2
* iommu_bind_table(dom2, ...) binds first-level pgtables for PASID #2

Quoted 1 line

Quoted 1 line

I'm trying to understand how to implement this with SMMU and other

This is proposed for architectures which support finer granularity second level translation with no impact on architectures which only support Source ID or the similar granularity.

Quoted 2 lines

IOMMUs. It's not a clean fit since we have a single domain to hold the second-level pgtables.

Do you mind explaining why a domain holds multiple second-level pgtables? Shouldn't that be multiple domains?

Quoted 9 lines

Then again, the nested case probably doesn't
matter for us - we might as well assign the parent directly, since all mdevs have the same second-level and can only be assigned to the same VM.

Also, can non-VFIO device drivers use auxiliary domains to do map/unmap on PASIDs? They are asking to do that and I'm proposing the private PASID thing, but since aux domains provide a similar feature we should probably converge somehow.

Yes, any non-VFIO device driver could use aux domain as well. The use model is:

iommu_enable_aux_domain(dev)
-- enables aux domain support for this device

iommu_domain_alloc(dev)
-- allocate an iommu domain

iommu_attach_device(domain, dev)
-- attach the domain to device

iommu_auxiliary_id(domain)
-- retrieve the pasid id used by this domain

The device driver then

iommu_map(domain, ...)

set the pasid id to hardware register and start to do dma.

From: Jean-Philippe Brucker
Sent: Thursday, September 13, 2018 1:54 AM

Quoted 16 lines · Lu Baolu

Hi,

Quoted 13 lines · Jean-Philippe Brucker

Hi,

Quoted 6 lines · Lu Baolu

Below APIs are introduced in the IOMMU glue for device drivers to use the finer granularity translation.

* iommu_capable(IOMMU_CAP_AUX_DOMAIN)
- Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware.

iommu_capable() cannot represent hardware capabilities, we need something else for systems with multiple IOMMUs that have different caps. How about iommu_domain_get_attr on the device's domain

instead?

Quoted 7 lines

Domain is not a good choice for per iommu cap query. A domain might be attached to devices belonging to different iommu's.

How about an API with device structure as parameter? A device always belongs to a specific iommu. This API is supposed to be used the device driver.

Ah right, domain attributes won't work. Your suggestion seems more suitable, but maybe users can simply try to enable auxiliary domains first, and conclude that the IOMMU doesn't support it if it returns an error

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

* iommu_en(dis)able_aux_domain(struct device *dev)
- Enable/disable the multiple domains capability for a device referenced by @dev.

It strikes me now that in the IOMMU driver, iommu_enable/disable_aux_domain() will do the same thing as iommu_sva_device_init/shutdown()
(https://www.spinics.net/lists/arm-kernel/msg651896.html). Some IOMMU drivers want to enable PASID and allocate PASID tables only when requested by users, in the sva_init_device IOMMU op (see Joerg's comment last year https://patchwork.kernel.org/patch/9989307/#21025429). Maybe we could simply add a flag to iommu_sva_device_init?

Quoted 13 lines

Quoted 7 lines

Quoted 6 lines

* iommu_auxiliary_id(struct iommu_domain *domain)
- Return the index value used for finer-granularity DMA translation. The specific device driver needs to feed the hardware with this value, so that hardware device could issue the DMA transaction with this value tagged.

Quoted 5 lines

This could also reuse iommu_domain_get_attr.

More generally I'm having trouble understanding how auxiliary domains will be used. So VFIO allocates PASIDs like this:

Quoted 7 lines

As I wrote in the cover letter, "auxiliary domain" is just a name to ease discussion. It's actually has no special meaning (we think a domain as an isolation boundary which could be used by the IOMMU to isolate the DMA transactions out of a PCI device or partial of it).

So drivers like vfio should see no difference when use an auxiliary domain. The auxiliary domain is not aware out of iommu driver.

For an auxiliary domain, VFIO does need to retrieve the PASID and write it to hardware. But being able to reuse iommu_map/unmap/iova_to_phys/etc
on the auxiliary domain is nice.

Quoted 10 lines

Quoted 10 lines

* iommu_enable_aux_domain(parent_dev)
* iommu_domain_alloc() -> dom1
* iommu_domain_alloc() -> dom2
* iommu_attach_device(dom1, parent_dev)
-> dom1 gets PASID #1
* iommu_attach_device(dom2, parent_dev)
-> dom2 gets PASID #2

Then I'm not sure about the next steps, when userspace does VFIO_IOMMU_MAP_DMA or VFIO_IOMMU_BIND on an mdev's

container. Is the

Quoted 15 lines

Quoted 15 lines

following use accurate?

For the single translation level:
* iommu_map(dom1, ...) updates first-level/second-level pgtables for PASID #1
* iommu_map(dom2, ...) updates first-level/second-level pgtables for PASID #2

Nested translation:
* iommu_map(dom1, ...) updates second-level pgtables for PASID #1
* iommu_bind_table(dom1, ...) binds first-level pgtables, provided by the guest, for PASID #1
* iommu_map(dom2, ...) updates second-level pgtables for PASID #2
* iommu_bind_table(dom2, ...) binds first-level pgtables for PASID #2

Quoted 1 line

Quoted 5 lines

Quoted 1 line

I'm trying to understand how to implement this with SMMU and other

This is proposed for architectures which support finer granularity second level translation with no impact on architectures which only support Source ID or the similar granularity.

Just to be clear, in this paragraph you're only referring to the Nested/second-level translation for mdev, which is specific to vt-d rev3? Other architectures can still do first-level translation with PASID, to support some use-cases of IOMMU aware mediated device (assigning mdevs to userspace drivers, for example)

Quoted 6 lines

Quoted 3 lines

IOMMUs. It's not a clean fit since we have a single domain to hold the second-level pgtables.

Do you mind explaining why a domain holds multiple second-level pgtables? Shouldn't that be multiple domains?

I didn't mean a single domain holding multiple second-level pgtables, but a single domain holding a single set of second-level pgtables for all mdevs. But let's ignore that, mdev and second-level isn't realistic for arm SMMU.

Quoted 4 lines

Quoted 4 lines

Then again, the nested case probably doesn't
matter for us - we might as well assign the parent directly, since all mdevs have the same second-level and can only be assigned to the same

VM.

Quoted 3 lines

Quoted 3 lines

Also, can non-VFIO device drivers use auxiliary domains to do

map/unmap

Quoted 19 lines

Quoted 3 lines

on PASIDs? They are asking to do that and I'm proposing the private PASID thing, but since aux domains provide a similar feature we should probably converge somehow.

Yes, any non-VFIO device driver could use aux domain as well. The use model is:

iommu_enable_aux_domain(dev)
-- enables aux domain support for this device

iommu_domain_alloc(dev)
-- allocate an iommu domain

iommu_attach_device(domain, dev)
-- attach the domain to device

iommu_auxiliary_id(domain)
-- retrieve the pasid id used by this domain

Quoted 5 lines

The device driver then

iommu_map(domain, ...)

set the pasid id to hardware register and start to do dma.

Sounds good, I'll drop the private PASID patch if we can figure out a solution to the attach/detach_dev problem discussed on patch 8/10

Quoted 9 lines

Quoted 3 lines

This is proposed for architectures which support finer granularity second level translation with no impact on architectures which only support Source ID or the similar granularity.

Just to be clear, in this paragraph you're only referring to the Nested/second-level translation for mdev, which is specific to vt-d rev3? Other architectures can still do first-level translation with PASID, to support some use-cases of IOMMU aware mediated device (assigning mdevs to userspace drivers, for example)

yes. aux domain concept applies only to vt-d rev3 which introduces scalable mode. Care is taken to avoid breaking usages on existing architectures.

one note. Assigning mdevs to user space alone doesn't imply IOMMU aware. All existing mdev usages use software or proprietary methods to isolate DMA. There is only one potential IOMMU aware mdev usage which we talked not rely on vt-d rev3 scalable mode - wrap a random PCI device into a single mdev instance (no sharing). In that case mdev inherits RID from parent PCI device, thus is isolated by IOMMU in RID granular. Our RFC supports this usage too. In VFIO two usages (PASID- based and RID-based) use same code path, i.e. always binding domain to the parent device of mdev. But within IOMMU they go different paths. PASID-based will go to aux-domain as iommu_enable_aux_domain has been called on that device. RID-based will follow existing unmanaged domain path, as if it is parent device assignment.

Quoted 3 lines

Sounds good, I'll drop the private PASID patch if we can figure out a solution to the attach/detach_dev problem discussed on patch 8/10

Can you elaborate a bit on private PASID usage? what is the high level flow on it?

Again based on earlier explanation, aux domain is specific to IOMMU architecture supporting vtd scalable mode-like capability, which allows separate 2nd/1st level translations per PASID. Need a better understanding how private PASID is relevant here.

Quoted 14 lines · Kevin

Quoted 9 lines

Quoted 3 lines

This is proposed for architectures which support finer granularity second level translation with no impact on architectures which only support Source ID or the similar granularity.

Just to be clear, in this paragraph you're only referring to the Nested/second-level translation for mdev, which is specific to vt-d rev3? Other architectures can still do first-level translation with PASID, to support some use-cases of IOMMU aware mediated device (assigning mdevs to userspace drivers, for example)

yes. aux domain concept applies only to vt-d rev3 which introduces scalable mode. Care is taken to avoid breaking usages on existing architectures.

Quoted 12 lines

one note. Assigning mdevs to user space alone doesn't imply IOMMU aware. All existing mdev usages use software or proprietary methods to isolate DMA. There is only one potential IOMMU aware mdev usage which we talked not rely on vt-d rev3 scalable mode - wrap a random PCI device into a single mdev instance (no sharing). In that case mdev inherits RID from parent PCI device, thus is isolated by IOMMU in RID granular. Our RFC supports this usage too. In VFIO two usages (PASID- based and RID-based) use same code path, i.e. always binding domain to the parent device of mdev. But within IOMMU they go different paths. PASID-based will go to aux-domain as iommu_enable_aux_domain has been called on that device. RID-based will follow existing unmanaged domain path, as if it is parent device assignment.

For Arm SMMU we're more interested in the PASID-granular case than the RID-granular one. It doesn't necessarily require vt-d rev3 scalable mode, the following example can be implemented with an SMMUv3, since it only needs PASID-granular first-level translation:

We have a PCI function that supports PASID, and can be partitioned into multiple isolated entities, mdevs. Each mdev has an MMIO frame, an MSI vector and a PASID.

Different processes (userspace drivers, not QEMU) each open one mdev. A process controlling one mdev has two ways of doing DMA:

(1) Classically, the process uses a VFIO_TYPE1v2_IOMMU container. This creates an auxiliary domain for the mdev, with PASID #35. The process creates DMA mappings with VFIO_IOMMU_MAP_DMA. VFIO calls iommu_map on the auxiliary domain. The IOMMU driver populates the pgtables associated with PASID #35.

(2) SVA. One way of doing it: the process uses a new "VFIO_TYPE1_SVA_IOMMU" type of container. VFIO binds the process address space to the device, gets PASID #35. Simpler, but not everyone wants to use SVA, especially not userspace drivers which need the highest performance.

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

So (2) would use iommu_sva_bind_device(), but (1) needs something else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

Another note: if for some reason you did want to allow userspace to choose between first-level or second-level, you could implement the VFIO_TYPE1_NESTING_IOMMU container. It acts like a VFIO_TYPE1v2_IOMMU, but also sets the DOMAIN_ATTR_NESTING on the IOMMU domain. So DMA_MAP ioctl on a NESTING container would populate second-level, and DMA_MAP on a normal container populates first-level. But if you're always going to use second-level by default, the distinction isn't necessary.

Quoted 6 lines

Quoted 5 lines

Sounds good, I'll drop the private PASID patch if we can figure out a solution to the attach/detach_dev problem discussed on patch 8/10

Quoted 7 lines

Can you elaborate a bit on private PASID usage? what is the high level flow on it?

Again based on earlier explanation, aux domain is specific to IOMMU architecture supporting vtd scalable mode-like capability, which allows separate 2nd/1st level translations per PASID. Need a better understanding how private PASID is relevant here.

Private PASIDs are used for doing iommu_map/iommu_unmap on PASIDs (first-level translation):
https://www.spinics.net/lists/dri-devel/msg177003.html As above, some people don't want SVA, some can't do it, some may even want a few private address spaces just for their kernel driver. They need a way to allocate PASIDs and do iommu_map/iommu_unmap on them, without binding to a process. I was planning to add the private PASID patch to my SVA series, but in my opinion the feature overlaps with auxiliary domains.

Quoted 4 lines

For Arm SMMU we're more interested in the PASID-granular case than the RID-granular one. It doesn't necessarily require vt-d rev3 scalable mode, the following example can be implemented with an SMMUv3, since it only needs PASID-granular first-level translation:

You are right, you can simply use the first level as IOVA for every PASID.

Only issue becomes when you need to assign that to a guest, you would be required to shadow the 1st level. If you have a 2nd level per-pasid first level can be managed in guest and don't require to shadow them.

Quoted 7 lines

Another note: if for some reason you did want to allow userspace to choose between first-level or second-level, you could implement the VFIO_TYPE1_NESTING_IOMMU container. It acts like a VFIO_TYPE1v2_IOMMU, but also sets the DOMAIN_ATTR_NESTING on the IOMMU domain. So DMA_MAP ioctl on a NESTING container would populate second-level, and DMA_MAP on a normal container populates first-level. But if you're always going to use second-level by default, the distinction isn't necessary.

Where is the nesting attribute specified? in vt-d2 it was part of context entry, so also meant all PASID's are nested now. In vt-d3 its part of PASID context.

It seems unsafe to share PASID's with different VM's since any request W/O PASID has only one mapping.

From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Thursday, September 13, 2018 11:03 PM

Quoted 14 lines · Kevin

Quoted 9 lines

Quoted 3 lines

This is proposed for architectures which support finer granularity second level translation with no impact on architectures which only support Source ID or the similar granularity.

Just to be clear, in this paragraph you're only referring to the Nested/second-level translation for mdev, which is specific to vt-d rev3? Other architectures can still do first-level translation with PASID, to support some use-cases of IOMMU aware mediated device (assigning mdevs to userspace drivers, for example)

yes. aux domain concept applies only to vt-d rev3 which introduces scalable mode. Care is taken to avoid breaking usages on existing architectures.

Quoted 12 lines

one note. Assigning mdevs to user space alone doesn't imply IOMMU aware. All existing mdev usages use software or proprietary methods to isolate DMA. There is only one potential IOMMU aware mdev usage which we talked not rely on vt-d rev3 scalable mode - wrap a random PCI device into a single mdev instance (no sharing). In that case mdev inherits RID from parent PCI device, thus is isolated by IOMMU in RID granular. Our RFC supports this usage too. In VFIO two usages (PASID- based and RID-based) use same code path, i.e. always binding domain to the parent device of mdev. But within IOMMU they go different paths. PASID-based will go to aux-domain as iommu_enable_aux_domain has been called on that device. RID-based will follow existing unmanaged domain path, as if it is parent device assignment.

For Arm SMMU we're more interested in the PASID-granular case than the RID-granular one. It doesn't necessarily require vt-d rev3 scalable mode, the following example can be implemented with an SMMUv3, since it
only needs PASID-granular first-level translation:

We have a PCI function that supports PASID, and can be partitioned into multiple isolated entities, mdevs. Each mdev has an MMIO frame, an MSI vector and a PASID.

Different processes (userspace drivers, not QEMU) each open one mdev. A process controlling one mdev has two ways of doing DMA:

(1) Classically, the process uses a VFIO_TYPE1v2_IOMMU container. This creates an auxiliary domain for the mdev, with PASID #35. The process creates DMA mappings with VFIO_IOMMU_MAP_DMA. VFIO calls iommu_map on
the auxiliary domain. The IOMMU driver populates the pgtables associated with PASID #35.

(2) SVA. One way of doing it: the process uses a new "VFIO_TYPE1_SVA_IOMMU" type of container. VFIO binds the process address
space to the device, gets PASID #35. Simpler, but not everyone wants to use SVA, especially not userspace drivers which need the highest performance.

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

So (2) would use iommu_sva_bind_device(), but (1) needs something else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

Another note: if for some reason you did want to allow userspace to choose between first-level or second-level, you could implement the VFIO_TYPE1_NESTING_IOMMU container. It acts like a VFIO_TYPE1v2_IOMMU,
but also sets the DOMAIN_ATTR_NESTING on the IOMMU domain. So DMA_MAP
ioctl on a NESTING container would populate second-level, and DMA_MAP on
a normal container populates first-level. But if you're always going to use second-level by default, the distinction isn't necessary.

Quoted 6 lines

Quoted 5 lines

Sounds good, I'll drop the private PASID patch if we can figure out a solution to the attach/detach_dev problem discussed on patch 8/10

Quoted 6 lines

Can you elaborate a bit on private PASID usage? what is the high level flow on it?

Again based on earlier explanation, aux domain is specific to IOMMU architecture supporting vtd scalable mode-like capability, which allows separate 2nd/1st level translations per PASID. Need a better

understanding

Quoted 1 line

how private PASID is relevant here.

Private PASIDs are used for doing iommu_map/iommu_unmap on PASIDs (first-level translation):
https://www.spinics.net/lists/dri-devel/msg177003.html As above, some people don't want SVA, some can't do it, some may even want a few private address spaces just for their kernel driver. They need a way to allocate PASIDs and do iommu_map/iommu_unmap on them, without binding to
a process. I was planning to add the private PASID patch to my SVA series, but in my opinion the feature overlaps with auxiliary domains.

Quoted 5 lines

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

the former yes applies to aux domain concept. The latter doesn't - you have only one second-level per device. whole PASID table managed by guest means you assign the whole device to guest, which is not the concept of aux domain here.

Quoted 9 lines

So (2) would use iommu_sva_bind_device(), but (1) needs something else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

Thanks for correcting me on this. You are right that aux domain shouldn't impose such limitation on 2nd or nested only. We define aux domain as a normal domain (aux takes effect only when attaching to a device), thus it should support all capabilities possible on a normal domain.

btw I'm not sure whether you look at my comment to patch 8/10. I explained the rationale why aux domain doesn't interfere with existing default domain usage, and in a quick thinking above example might not make difference. but need your confirm here. :-)

Quoted 14 lines · Kevin

Quoted 9 lines

Quoted 3 lines

This is proposed for architectures which support finer granularity second level translation with no impact on architectures which only support Source ID or the similar granularity.

Just to be clear, in this paragraph you're only referring to the Nested/second-level translation for mdev, which is specific to vt-d rev3? Other architectures can still do first-level translation with PASID, to support some use-cases of IOMMU aware mediated device (assigning mdevs to userspace drivers, for example)

yes. aux domain concept applies only to vt-d rev3 which introduces scalable mode. Care is taken to avoid breaking usages on existing architectures.

Quoted 13 lines

one note. Assigning mdevs to user space alone doesn't imply IOMMU aware. All existing mdev usages use software or proprietary methods to isolate DMA. There is only one potential IOMMU aware mdev usage which we talked not rely on vt-d rev3 scalable mode - wrap a random PCI device into a single mdev instance (no sharing). In that case mdev inherits RID from parent PCI device, thus is isolated by IOMMU in RID granular. Our RFC supports this usage too. In VFIO two usages (PASID- based and RID-based) use same code path, i.e. always binding domain to the parent device of mdev. But within IOMMU they go different paths. PASID-based will go to aux-domain as iommu_enable_aux_domain has been called on that device. RID-based will follow existing unmanaged domain path, as if it is parent device assignment.

For Arm SMMU we're more interested in the PASID-granular case than the RID-granular one. It doesn't necessarily require vt-d rev3 scalable mode, the following example can be implemented with an SMMUv3, since it only needs PASID-granular first-level translation:

We have a PCI function that supports PASID, and can be partitioned into multiple isolated entities, mdevs. Each mdev has an MMIO frame, an MSI vector and a PASID.

Different processes (userspace drivers, not QEMU) each open one mdev. A process controlling one mdev has two ways of doing DMA:

(1) Classically, the process uses a VFIO_TYPE1v2_IOMMU container. This creates an auxiliary domain for the mdev, with PASID #35. The process creates DMA mappings with VFIO_IOMMU_MAP_DMA. VFIO calls iommu_map on the auxiliary domain. The IOMMU driver populates the pgtables associated with PASID #35.

(2) SVA. One way of doing it: the process uses a new "VFIO_TYPE1_SVA_IOMMU" type of container. VFIO binds the process address space to the device, gets PASID #35. Simpler, but not everyone wants to use SVA, especially not userspace drivers which need the highest performance.

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

So (2) would use iommu_sva_bind_device(),

but (1) needs something
else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

Another note: if for some reason you did want to allow userspace to choose between first-level or second-level, you could implement the VFIO_TYPE1_NESTING_IOMMU container. It acts like a VFIO_TYPE1v2_IOMMU, but also sets the DOMAIN_ATTR_NESTING on the IOMMU domain. So DMA_MAP ioctl on a NESTING container would populate second-level, and DMA_MAP on a normal container populates first-level. But if you're always going to use second-level by default, the distinction isn't necessary.

Quoted 5 lines

Quoted 4 lines

Sounds good, I'll drop the private PASID patch if we can figure out a solution to the attach/detach_dev problem discussed on patch 8/10

Quoted 7 lines

Can you elaborate a bit on private PASID usage? what is the high level flow on it?

Again based on earlier explanation, aux domain is specific to IOMMU architecture supporting vtd scalable mode-like capability, which allows separate 2nd/1st level translations per PASID. Need a better understanding how private PASID is relevant here.

Private PASIDs are used for doing iommu_map/iommu_unmap on PASIDs (first-level translation):
https://www.spinics.net/lists/dri-devel/msg177003.html As above, some people don't want SVA, some can't do it, some may even want a few private address spaces just for their kernel driver. They need a way to allocate PASIDs and do iommu_map/iommu_unmap on them, without binding to a process. I was planning to add the private PASID patch to my SVA series, but in my opinion the feature overlaps with auxiliary domains.

Thanks,
Jean
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Quoted 6 lines

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

There is a difference in case of guest SVA. VT-d v3 will bind guest PASID and guest CR3 instead of the guest PASID table. Then turn on nesting. In case of mdev, the second level is obtained from the aux domain which was setup for the default PASID. Or in case of PCI device, second level is harvested from RID2PASID.

Quoted 1 line

So (2) would use iommu_sva_bind_device(),

We would need something different than that for guest bind, just to show the two cases:>
int iommu_sva_bind_device(struct device *dev, struct mm_struct *mm, int *pasid, unsigned long flags, void *drvdata)

(WIP)
int sva_bind_gpasid(struct device *dev, struct gpasid_bind_data *data) where:
/**
 * struct gpasid_bind_data - Information about device and guest PASID binding
 * @pasid:       Process address space ID used for the guest mm
 * @addr_width:  Guest address width. Paging mode can also be derived.
 * @gcr3:        Guest CR3 value from guest mm
 */
struct gpasid_bind_data {
        __u32 pasid;

        __u64 gcr3;
        __u32 addr_width;
        __u32 flags;
#define IOMMU_SVA_GPASID_SRE    BIT(0) /* supervisor request */ };
Perhaps there is room to merge with io_mm but the life cycle management of guest PASID and host PASID will be different if you rely on mm release callback than FD.

Quoted 10 lines

but (1) needs something
else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

I think the intention is that when a mdev is created, we don;t know whether it will be used for SVA or IOVA. So aux domain is here to "hold a spot" for the default PASID such that MAP_DMA calls can work as usual, which is second level only. Later, if SVA is used on the mdev there will be another PASID allocated for that purpose.
Do we need to create an aux domain for each PASID? the translation can be looked up by the combination of parent dev and pasid.

From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Tuesday, September 18, 2018 11:47 PM

Quoted 11 lines · Jacob Pan

Quoted 6 lines

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

There is a difference in case of guest SVA. VT-d v3 will bind guest PASID and guest CR3 instead of the guest PASID table. Then turn on nesting. In case of mdev, the second level is obtained from the aux domain which was setup for the default PASID. Or in case of PCI device, second level is harvested from RID2PASID.

Right, though I wasn't talking about the host managing guest SVA here, but a kernel binding the address space of one of its userspace drivers to the mdev.

Quoted 4 lines

Quoted 1 line

So (2) would use iommu_sva_bind_device(),

We would need something different than that for guest bind, just to show the two cases:>
int iommu_sva_bind_device(struct device *dev, struct mm_struct *mm,

int

Quoted 12 lines

*pasid, unsigned long flags, void *drvdata)

(WIP)
int sva_bind_gpasid(struct device *dev, struct gpasid_bind_data *data) where:
/**
 * struct gpasid_bind_data - Information about device and guest PASID binding
 * @pasid:       Process address space ID used for the guest mm
 * @addr_width:  Guest address width. Paging mode can also be derived.
 * @gcr3:        Guest CR3 value from guest mm
 */

Quoted 8 lines

struct gpasid_bind_data {
        __u32 pasid;
        __u64 gcr3;
        __u32 addr_width;
        __u32 flags;
#define IOMMU_SVA_GPASID_SRE    BIT(0) /* supervisor request */ };
Perhaps there is room to merge with io_mm but the life cycle

management

Quoted 2 lines

of guest PASID and host PASID will be different if you rely on mm release callback than FD.

I think gpasid management should stay separate from io_mm, since in your case VFIO mechanisms are used for life cycle management of the VM, similarly to the former bind_pasid_table proposal. For example closing the container fd would unbind all guest page tables. The QEMU process' address space lifetime seems like the wrong thing to track for gpasid.

Quoted 11 lines

Quoted 11 lines

but (1) needs something
else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

Quoted 7 lines

I think the intention is that when a mdev is created, we don;t know whether it will be used for SVA or IOVA. So aux domain is here to "hold a spot" for the default PASID such that MAP_DMA calls can work as usual, which is second level only. Later, if SVA is used on the mdev there will be another PASID allocated for that purpose.
Do we need to create an aux domain for each PASID? the translation can be looked up by the combination of parent dev and pasid.

When allocating a new PASID for the guest, I suppose you need to clone the second-level translation config? In which case a single aux domain for the mdev might be easier to implement in the IOMMU driver. Entirely up to you since we don't have this case on SMMUv3

Quoted 48 lines

From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Tuesday, September 18, 2018 11:47 PM

Quoted 11 lines · Jacob Pan

Quoted 6 lines

This example only needs to modify first-level translation, and works with SMMUv3. The kernel here could be the host, in which case second-level translation is disabled in the SMMU, or it could be the guest, in which case second-level mappings are created by QEMU and first-level translation is managed by assigning PASID tables to the guest.

There is a difference in case of guest SVA. VT-d v3 will bind guest PASID and guest CR3 instead of the guest PASID table. Then turn on nesting. In case of mdev, the second level is obtained from the aux domain which was setup for the default PASID. Or in case of PCI device, second level is harvested from RID2PASID.

Right, though I wasn't talking about the host managing guest SVA here, but a kernel binding the address space of one of its userspace drivers to the mdev.

Quoted 5 lines

Quoted 1 line

So (2) would use iommu_sva_bind_device(),

We would need something different than that for guest bind, just to show the two cases:>
int iommu_sva_bind_device(struct device *dev, struct mm_struct *mm,

int

Quoted 13 lines

*pasid, unsigned long flags, void *drvdata)

(WIP)
int sva_bind_gpasid(struct device *dev, struct gpasid_bind_data *data) where:
/**
 * struct gpasid_bind_data - Information about device and guest PASID binding
 * @pasid:       Process address space ID used for the guest mm
 * @addr_width:  Guest address width. Paging mode can also be derived.
 * @gcr3:        Guest CR3 value from guest mm
 */

Quoted 8 lines

struct gpasid_bind_data {
        __u32 pasid;
        __u64 gcr3;
        __u32 addr_width;
        __u32 flags;
#define IOMMU_SVA_GPASID_SRE    BIT(0) /* supervisor request */ };
Perhaps there is room to merge with io_mm but the life cycle

management

Quoted 2 lines

Quoted 2 lines

of guest PASID and host PASID will be different if you rely on mm release callback than FD.

let's not calling gpasid here - which makes sense only in bind_pasid_table proposal where pasid table thus pasid space is managed by guest. In above context it is always about host pasid (allocated in system-wide), which could point to a host cr3 (user process) or a guest cr3 (vm case).

Quoted 6 lines

I think gpasid management should stay separate from io_mm, since in your case VFIO mechanisms are used for life cycle management of the VM, similarly to the former bind_pasid_table proposal. For example closing the container fd would unbind all guest page tables. The QEMU process' address space lifetime seems like the wrong thing to track for gpasid.

I sort of agree (though not thinking through all the flow carefully). PASIDs are allocated per iommu domain, thus release also happens when domain is detached (along with container fd close).

Quoted 20 lines

Quoted 11 lines

Quoted 11 lines

but (1) needs something
else. Aren't auxiliary domains suitable for (1)? Why limit auxiliary domain to second-level or nested translation? It seems silly to use a different API for first-level, since the flow in userspace and VFIO is the same as your second-level case as far as MAP_DMA ioctl goes. The difference is that in your case the auxiliary domain supports an additional operation which binds first-level page tables. An auxiliary domain that only supports first-level wouldn't support this operation, but it can still implement iommu_map/unmap/etc.

Quoted 8 lines

I think the intention is that when a mdev is created, we don;t know whether it will be used for SVA or IOVA. So aux domain is here to "hold a spot" for the default PASID such that MAP_DMA calls can work as usual, which is second level only. Later, if SVA is used on the mdev there will be another PASID allocated for that purpose. Do we need to create an aux domain for each PASID? the translation can be looked up by the combination of parent dev and pasid.

Quoted 5 lines

When allocating a new PASID for the guest, I suppose you need to clone the second-level translation config? In which case a single aux domain for the mdev might be easier to implement in the IOMMU driver. Entirely up to you since we don't have this case on SMMUv3

One thing to highlight in related discussions (also mentioned in other thread). There is not a new iommu domain type called 'aux'. 'aux' matters only to a specific device when a domain is attached to that device which has aux capability enabled. Same domain can be attached to other device as normal domain. In that case multiple PASIDs allocated on same mdev are tied to same aux domain, same bare metal SVA case, i.e. any domain (normal or aux) can include 2nd level structure and multiple 1st level structures. Jean is correct - all PASIDs in same domain then share 2nd level translation, and there are io_mm or similar tracking structures to associate each PASID to a 1st level translation structure.

Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Quoted 18 lines · Lu Baolu

Hi,

Quoted 6 lines · Jean-Philippe Brucker

Hi,

Quoted 3 lines · Lu Baolu

Below APIs are introduced in the IOMMU glue for device drivers to use the finer granularity translation.

Quoted 6 lines

Quoted 3 lines

* iommu_capable(IOMMU_CAP_AUX_DOMAIN)
- Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware.

iommu_capable() cannot represent hardware capabilities, we need something else for systems with multiple IOMMUs that have different caps. How about iommu_domain_get_attr on the device's domain instead?

Domain is not a good choice for per iommu cap query. A domain might be attached to devices belonging to different iommu's.

Quoted 3 lines

How about an API with device structure as parameter? A device always belongs to a specific iommu. This API is supposed to be used the device driver.

Ah right, domain attributes won't work. Your suggestion seems more suitable, but maybe users can simply try to enable auxiliary domains first, and conclude that the IOMMU doesn't support it if it returns an error

From: Lu Baolu [mailto:baolu.lu-VuQAYsv1563Yd54FQh9/[email protected]]
Sent: Friday, September 14, 2018 10:47 AM

Hi,

Quoted 8 lines · Jean-Philippe Brucker

Quoted 7 lines · Lu Baolu

Hi,

Quoted 4 lines · Jean-Philippe Brucker

Hi,

Quoted 1 line · Lu Baolu

Below APIs are introduced in the IOMMU glue for device drivers to

use

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 5 lines

the finer granularity translation.

* iommu_capable(IOMMU_CAP_AUX_DOMAIN)
- Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware.

iommu_capable() cannot represent hardware capabilities, we need something else for systems with multiple IOMMUs that have different caps. How about iommu_domain_get_attr on the device's domain

instead?

Quoted 1 line

Quoted 1 line

Domain is not a good choice for per iommu cap query. A domain might

be

Quoted 8 lines

Quoted 5 lines

attached to devices belonging to different iommu's.

How about an API with device structure as parameter? A device always belongs to a specific iommu. This API is supposed to be used the device driver.

Ah right, domain attributes won't work. Your suggestion seems more suitable, but maybe users can simply try to enable auxiliary domains first, and conclude that the IOMMU doesn't support it if it returns an

error

Quoted 1 line

Some driver might want to check whether hardware supports AUX_DOMAIN
during the driver probe stage, but doesn't want to enable AUX_DOMAIN at that time. One reasonable use case is driver check AUX_DOMAIN cap during driver probe and expose different sysfs nodes according to whether AUX_DOMAIN is support or not, then AUX_DOMAIN is enabled or disabled during run time through a sysfs node. With this consideration, we still need a API to check cap.

How about

* iommu_check_aux_domain(struct device *dev)
- Check whether the iommu driver supports multiple domains on @dev.