Define EINVAL as device/domain incompatibility

Cases like VFIO wish to attach a device to an existing domain that was not allocated specifically from the device. This raises a condition where the IOMMU driver can fail the domain attach because the domain and device are incompatible with each other.

This is a soft failure that can be resolved by using a different domain.

Provide a dedicated errno EINVAL from the IOMMU driver during attach that the reason why the attach failed is because of domain incompatibility.

VFIO can use this to know that the attach is a soft failure and it should continue searching. Otherwise, the attach will be a hard failure and VFIO will return the code to userspace.

Update kdocs to add rules of return value to the attach_dev op and APIs.

Suggested-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Signed-off-by Nicolin Chen <[email protected]>

---
  drivers/iommu/iommu.c | 24 ++++++++++++++++++++++++
  include/linux/iommu.h | 12 ++++++++++++
  2 files changed, 36 insertions(+)

drivers/iommu/iommu.c +24 -0

index 4893c2429ca5..3c0c5d64bb50 100644
@@ -1949,6 +1949,18 @@ static int __iommu_attach_device(struct iommu_domain *domain,
  	return ret;
  }
 
+/**
+ * iommu_attach_device - Attach an IOMMU domain to a device
+ * @domain: IOMMU domain to attach
+ * @dev: Device that will be attached
+ *
+ * Returns 0 on success and error code on failure
+ *
+ * Note that EINVAL can be treated as a soft failure, indicating
+ * that certain configuration of the domain is incompatible with
+ * the device. In this case attaching a different domain to the
+ * device may succeed.
+ */
  int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
  {
  	struct iommu_group *group;

@@ -2075,6 +2087,18 @@ static int __iommu_attach_group(struct iommu_domain *domain,
  	return ret;
  }
 
+/**
+ * iommu_attach_group - Attach an IOMMU domain to an IOMMU group
+ * @domain: IOMMU domain to attach
+ * @group: IOMMU group that will be attached
+ *
+ * Returns 0 on success and error code on failure
+ *
+ * Note that EINVAL can be treated as a soft failure, indicating
+ * that certain configuration of the domain is incompatible with
+ * the group. In this case attaching a different domain to the
+ * group may succeed.
+ */
  int iommu_attach_group(struct iommu_domain *domain, struct iommu_group *group)
  {
  	int ret;

include/linux/iommu.h +5 -0

index a325532aeab5..6f3569340e8a 100644
@@ -266,6 +266,18 @@ struct iommu_ops {
  /**
   * struct iommu_domain_ops - domain specific operations
   * @attach_dev: attach an iommu domain to a device
+ *  Return:
+ * * 0		- success
+ * * EINVAL	- can indicate that device and domain are incompatible due to
+ *		  some previous configuration of the domain, in which case the
+ *		  driver shouldn't log an error, since it is legitimate for a

Following the new rules in include/linux/iommu.h kdocs, update all drivers ->attach_dev callback functions to return EINVAL in the failure paths that are related to domain incompatibility.

Also, drop adjacent error prints to prevent a kernel log spam.

Reviewed-by Jean-Philippe Brucker <[email protected]>

Reviewed-by Lu Baolu <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Signed-off-by Nicolin Chen <[email protected]>

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 +----------
 drivers/iommu/arm/arm-smmu/arm-smmu.c       |  3 ---
 drivers/iommu/arm/arm-smmu/qcom_iommu.c     |  7 +------
 drivers/iommu/intel/iommu.c                 | 10 +++-------
 drivers/iommu/ipmmu-vmsa.c                  |  2 --
 drivers/iommu/omap-iommu.c                  |  2 +-
 drivers/iommu/sprd-iommu.c                  |  4 +---
 drivers/iommu/tegra-gart.c                  |  2 +-
 drivers/iommu/virtio-iommu.c                |  3 +--
 9 files changed, 9 insertions(+), 35 deletions(-)

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -7

index ba47c73f5b8c..01fd7df16cb9 100644
@@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 			goto out_unlock;
 		}
 	} else if (smmu_domain->smmu != smmu) {
-		dev_err(dev,
-			"cannot attach to SMMU %s (upstream of %s)\n",
-			dev_name(smmu_domain->smmu->dev),
-			dev_name(smmu->dev));
-		ret = -ENXIO;
+		ret = -EINVAL;
 		goto out_unlock;
 	} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
 		   master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
-		dev_err(dev,
-			"cannot attach to incompatible domain (%u SSID bits != %u)\n",

Quoted 25 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -9

index ba47c73f5b8c..01fd7df16cb9 100644
@@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
                      goto out_unlock;
              }
      } else if (smmu_domain->smmu != smmu) {
-             dev_err(dev,
-                     "cannot attach to SMMU %s (upstream of %s)\n",
-                     dev_name(smmu_domain->smmu->dev),
-                     dev_name(smmu->dev));
-             ret = -ENXIO;
+             ret = -EINVAL;
              goto out_unlock;
      } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
                 master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
-             dev_err(dev,
-                     "cannot attach to incompatible domain (%u SSID bits != %u)\n",
-                     smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
              ret = -EINVAL;
              goto out_unlock;
      } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
                 smmu_domain->stall_enabled != master->stall_enabled) {
-             dev_err(dev, "cannot attach to stall-%s domain\n",

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 2443

Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -1

-                     smmu_domain->stall_enabled ? "enabled" : "disabled");
              ret = -EINVAL;
              goto out_unlock;
      }

I think it would be helpful to preserve these messages using dev_err_ratelimited() so that attach failure can be diagnosed without having to hack the messages back into the driver.

Quoted 25 lines · Will Deacon

Quoted 25 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -9

index ba47c73f5b8c..01fd7df16cb9 100644
@@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
                      goto out_unlock;
              }
      } else if (smmu_domain->smmu != smmu) {
-             dev_err(dev,
-                     "cannot attach to SMMU %s (upstream of %s)\n",
-                     dev_name(smmu_domain->smmu->dev),
-                     dev_name(smmu->dev));
-             ret = -ENXIO;
+             ret = -EINVAL;
              goto out_unlock;
      } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
                 master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
-             dev_err(dev,
-                     "cannot attach to incompatible domain (%u SSID bits != %u)\n",
-                     smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
              ret = -EINVAL;
              goto out_unlock;
      } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
                 smmu_domain->stall_enabled != master->stall_enabled) {
-             dev_err(dev, "cannot attach to stall-%s domain\n",

Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 2443

Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -1

-                     smmu_domain->stall_enabled ? "enabled" : "disabled");
              ret = -EINVAL;
              goto out_unlock;
      }

I think it would be helpful to preserve these messages using dev_err_ratelimited() so that attach failure can be diagnosed without having to hack the messages back into the driver.

Thank you for the review.

The change is already picked up last week. Yet, I can add prints back with a followup patch, if no one has a problem with that.

Also, I am not quite sure what the use case would be to have an error print. Perhaps dev_dbg() would be more fitting if it is just for diagnosis?

Quoted 36 lines · Nicolin Chen

Quoted 25 lines · Will Deacon

Quoted 25 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -9

index ba47c73f5b8c..01fd7df16cb9 100644
@@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
                      goto out_unlock;
              }
      } else if (smmu_domain->smmu != smmu) {
-             dev_err(dev,
-                     "cannot attach to SMMU %s (upstream of %s)\n",
-                     dev_name(smmu_domain->smmu->dev),
-                     dev_name(smmu->dev));
-             ret = -ENXIO;
+             ret = -EINVAL;
              goto out_unlock;
      } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
                 master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
-             dev_err(dev,
-                     "cannot attach to incompatible domain (%u SSID bits != %u)\n",
-                     smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
              ret = -EINVAL;
              goto out_unlock;
      } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
                 smmu_domain->stall_enabled != master->stall_enabled) {
-             dev_err(dev, "cannot attach to stall-%s domain\n",

Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 2443

Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -1

-                     smmu_domain->stall_enabled ? "enabled" : "disabled");
              ret = -EINVAL;
              goto out_unlock;
      }

I think it would be helpful to preserve these messages using dev_err_ratelimited() so that attach failure can be diagnosed without having to hack the messages back into the driver.

Quoted 4 lines

Thank you for the review.

The change is already picked up last week. Yet, I can add prints back with a followup patch, if no one has a problem with that.

Sorry, I fell behind with upstream so I got to this late. A patch on top would be fantastic!

Quoted 3 lines

Also, I am not quite sure what the use case would be to have an error print. Perhaps dev_dbg() would be more fitting if it is just for diagnosis?

Sure, that works for me. I think the messages are useful for folks triggering this path e.g. via sysfs but if they're limited to debug I think that's better than removing them altogether.

This series is to replace the previous EMEDIUMTYPE patch in a VFIO series:
https://lore.kernel.org/kvm/[email protected]/

The purpose is to regulate all existing ->attach_dev callback functions to use EINVAL exclusively for an incompatibility error between a device and a domain. This allows VFIO and IOMMUFD to detect such a soft error, and then try a different domain with the same device.

Among all the patches, the first two are preparatory changes. And then one patch to update kdocs and another three patches for the enforcement effort.

Although it might be ideal to merge the previous VFIO series together with this series, given the number of new changes, the review in the IOMMU list might need a couple of rounds to finalize. Also, considering that v6.0 is at rc5 now, perhaps we could merge this IOMMU series and the VFIO one in different cycles to avoid merge conflicts. If there's less concern for it, I can respin the finalized version of this series with the previous VFIO one to merge together into the VFIO tree.

This series is also available on Github:
https://github.com/nicolinc/iommufd/commits/iommu_attach_dev-v7