Skip to contentSkip to conversation

iommu: Prepare to deliver page faults to user space

PATCH Superseded Lu Baolu 4 participants 25 messages Sep 14, 2023 – Sep 26, 2023

Subsystems (5)

v5 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Lu Baolu

author maintainer
#0 lore raw

v5 cover iommu: Prepare to deliver page faults to user space

When a user-managed page table is attached to an IOMMU, it is necessary to deliver IO page faults to user space so that they can be handled appropriately. One use case for this is nested translation, which is currently being discussed in the mailing list.

I have posted a RFC series [1] that describes the implementation of delivering page faults to user space through IOMMUFD. This series has received several comments on the IOMMU refactoring, which I am trying to address in this series.

The major refactoring includes:

- [PATCH 01 ~ 04] Move include/uapi/linux/iommu.h to include/linux/iommu.h. Remove the unrecoverable fault data definition.
- [PATCH 05 ~ 06] Remove iommu_[un]register_device_fault_handler().
- [PATCH 07 ~ 10] Separate SVA and IOPF. Make IOPF a generic page fault handling framework.
- [PATCH 11 ~ 12] Improve iopf framework for iommufd use.

This is also available at github [2]. I would appreciate your feedback on this series.

[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://github.com/LuBaolu/intel-iommu/commits/preparatory-io-pgfault-delivery-v5

Change log:
v5:
- Consolidate per-device fault data management. (New patch 11)
- Improve iopf_queue_flush_dev(). (New patch 12)

v4: https://lore.kernel.org/linux-iommu/[email protected]/
- Merge iommu_fault_event and iopf_fault. They are duplicate.
- Move iommu_report_device_fault() and iommu_page_response() to io-pgfault.c.
- Move iommu_sva_domain_alloc() to iommu-sva.c.
- Add group->domain and use it directly in sva fault handler.
- Misc code refactoring and refining.

v3: https://lore.kernel.org/linux-iommu/[email protected]/
- Convert the fault data structures from uAPI to kAPI.
- Merge iopf_device_param into iommu_fault_param.
- Add debugging on domain lifetime for iopf.
- Remove patch "iommu: Change the return value of dev_iommu_get()".
- Remove patch "iommu: Add helper to set iopf handler for domain".
- Misc code refactoring and refining.

v2: https://lore.kernel.org/linux-iommu/[email protected]/
- Remove unrecoverable fault data definition as suggested by Kevin.
- Drop the per-device fault cookie code considering that doesn't make
much sense for SVA.
- Make the IOMMU page fault handling framework generic. So that it can available for use cases other than SVA.

v1: https://lore.kernel.org/linux-iommu/[email protected]/

Lu Baolu (12):
iommu: Move iommu fault data to linux/iommu.h iommu/arm-smmu-v3: Remove unrecoverable faults reporting
iommu: Remove unrecoverable fault data
iommu: Cleanup iopf data structure definitions
iommu: Merge iopf_device_param into iommu_fault_param
iommu: Remove iommu_[un]register_device_fault_handler()
iommu: Merge iommu_fault_event and iopf_fault
iommu: Prepare for separating SVA and IOPF
iommu: Make iommu_queue_iopf() more generic
iommu: Separate SVA and IOPF
iommu: Consolidate per-device fault data management
iommu: Improve iopf_queue_flush_dev()

 include/linux/iommu.h                         | 258 ++++++++---
 drivers/iommu/intel/iommu.h                   |   2 +-
 drivers/iommu/iommu-sva.h                     |  71 ---
 include/uapi/linux/iommu.h                    | 161 -------
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |  14 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  51 +--
 drivers/iommu/intel/iommu.c                   |  25 +-
 drivers/iommu/intel/svm.c                     |   8 +-
 drivers/iommu/io-pgfault.c                    | 424 ++++++++++++------
 drivers/iommu/iommu-sva.c                     |  82 +++-
 drivers/iommu/iommu.c                         | 233 ----------
 MAINTAINERS                                   |   1 -
 drivers/iommu/Kconfig                         |   4 +
 drivers/iommu/Makefile                        |   3 +-
 drivers/iommu/intel/Kconfig                   |   1 +
 15 files changed, 571 insertions(+), 767 deletions(-)
 delete mode 100644 drivers/iommu/iommu-sva.h
 delete mode 100644 include/uapi/linux/iommu.h
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v5 1/12 iommu: Move iommu fault data to linux/iommu.h

The iommu fault data is currently defined in uapi/linux/iommu.h, but is only used inside the iommu subsystem. Move it to linux/iommu.h, where it will be more accessible to kernel drivers.

With this done, uapi/linux/iommu.h becomes empty and can be removed from the tree.

---
 include/linux/iommu.h      | 152 +++++++++++++++++++++++++++++++++-
 include/uapi/linux/iommu.h | 161 -------------------------------------
 MAINTAINERS                |   1 -
 3 files changed, 151 insertions(+), 163 deletions(-)
 delete mode 100644 include/uapi/linux/iommu.h
include/linux/iommu.h +151 -1
index c50a769d569a..895405567f86 100644
@@ -13,7 +13,6 @@
 #include <linux/errno.h>
 #include <linux/err.h>
 #include <linux/of.h>
-#include <uapi/linux/iommu.h>
 
 #define IOMMU_READ	(1 << 0)
 #define IOMMU_WRITE	(1 << 1)
@@ -42,6 +41,157 @@ struct iommu_sva;
 struct iommu_fault_event;
 struct iommu_dma_cookie;
 
+#define IOMMU_FAULT_PERM_READ	(1 << 0) /* read */
+#define IOMMU_FAULT_PERM_WRITE	(1 << 1) /* write */
+#define IOMMU_FAULT_PERM_EXEC	(1 << 2) /* exec */
+#define IOMMU_FAULT_PERM_PRIV	(1 << 3) /* privileged */
+
+/* Generic fault types, can be expanded IRQ remapping fault */
+enum iommu_fault_type {
+	IOMMU_FAULT_DMA_UNRECOV = 1,	/* unrecoverable fault */
+	IOMMU_FAULT_PAGE_REQ,		/* page request fault */
+};
+
+enum iommu_fault_reason {
+	IOMMU_FAULT_REASON_UNKNOWN = 0,
+
+	/* Could not access the PASID table (fetch caused external abort) */
+	IOMMU_FAULT_REASON_PASID_FETCH,
+
+	/* PASID entry is invalid or has configuration errors */
+	IOMMU_FAULT_REASON_BAD_PASID_ENTRY,
+
+	/*
+	 * PASID is out of range (e.g. exceeds the maximum PASID
+	 * supported by the IOMMU) or disabled.
+	 */
+	IOMMU_FAULT_REASON_PASID_INVALID,
+
+	/*
+	 * An external abort occurred fetching (or updating) a translation
+	 * table descriptor
+	 */
+	IOMMU_FAULT_REASON_WALK_EABT,
+
+	/*
+	 * Could not access the page table entry (Bad address),
+	 * actual translation fault
+	 */
+	IOMMU_FAULT_REASON_PTE_FETCH,
+
+	/* Protection flag check failed */
+	IOMMU_FAULT_REASON_PERMISSION,
+
+	/* access flag check failed */
+	IOMMU_FAULT_REASON_ACCESS,
+
+	/* Output address of a translation stage caused Address Size fault */
+	IOMMU_FAULT_REASON_OOR_ADDRESS,
+};
+
+/**
+ * struct iommu_fault_unrecoverable - Unrecoverable fault data
+ * @reason: reason of the fault, from &enum iommu_fault_reason
+ * @flags: parameters of this fault (IOMMU_FAULT_UNRECOV_* values)
+ * @pasid: Process Address Space ID
+ * @perm: requested permission access using by the incoming transaction
+ *        (IOMMU_FAULT_PERM_* values)
+ * @addr: offending page address
+ * @fetch_addr: address that caused a fetch abort, if any
+ */
+struct iommu_fault_unrecoverable {
+	__u32	reason;
+#define IOMMU_FAULT_UNRECOV_PASID_VALID		(1 << 0)
+#define IOMMU_FAULT_UNRECOV_ADDR_VALID		(1 << 1)
+#define IOMMU_FAULT_UNRECOV_FETCH_ADDR_VALID	(1 << 2)
+	__u32	flags;
+	__u32	pasid;
+	__u32	perm;
+	__u64	addr;
+	__u64	fetch_addr;
+};
+
+/**
+ * struct iommu_fault_page_request - Page Request data
+ * @flags: encodes whether the corresponding fields are valid and whether this
+ *         is the last page in group (IOMMU_FAULT_PAGE_REQUEST_* values).
+ *         When IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID is set, the page response
+ *         must have the same PASID value as the page request. When it is clear,
+ *         the page response should not have a PASID.
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: requested page permissions (IOMMU_FAULT_PERM_* values)
+ * @addr: page address
+ * @private_data: device-specific private information
+ */
+struct iommu_fault_page_request {
+#define IOMMU_FAULT_PAGE_REQUEST_PASID_VALID	(1 << 0)
+#define IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE	(1 << 1)
+#define IOMMU_FAULT_PAGE_REQUEST_PRIV_DATA	(1 << 2)
+#define IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID	(1 << 3)
+	__u32	flags;
+	__u32	pasid;
+	__u32	grpid;
+	__u32	perm;
+	__u64	addr;
+	__u64	private_data[2];
+};
+
+/**
+ * struct iommu_fault - Generic fault data
+ * @type: fault type from &enum iommu_fault_type
+ * @padding: reserved for future use (should be zero)
+ * @event: fault event, when @type is %IOMMU_FAULT_DMA_UNRECOV
+ * @prm: Page Request message, when @type is %IOMMU_FAULT_PAGE_REQ
+ * @padding2: sets the fault size to allow for future extensions
+ */
+struct iommu_fault {
+	__u32	type;
+	__u32	padding;
+	union {
+		struct iommu_fault_unrecoverable event;
+		struct iommu_fault_page_request prm;
+		__u8 padding2[56];
+	};
+};
+
+/**
+ * enum iommu_page_response_code - Return status of fault handlers
+ * @IOMMU_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ *	populated, retry the access. This is "Success" in PCI PRI.
+ * @IOMMU_PAGE_RESP_FAILURE: General error. Drop all subsequent faults from
+ *	this device if possible. This is "Response Failure" in PCI PRI.
+ * @IOMMU_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ *	access. This is "Invalid Request" in PCI PRI.
+ */
+enum iommu_page_response_code {
+	IOMMU_PAGE_RESP_SUCCESS = 0,
+	IOMMU_PAGE_RESP_INVALID,
+	IOMMU_PAGE_RESP_FAILURE,
+};
+
+/**
+ * struct iommu_page_response - Generic page response information
+ * @argsz: User filled size of this data
+ * @version: API version of this structure
+ * @flags: encodes whether the corresponding fields are valid
+ *         (IOMMU_FAULT_PAGE_RESPONSE_* values)
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @code: response code from &enum iommu_page_response_code
+ */
+struct iommu_page_response {
+	__u32	argsz;
+#define IOMMU_PAGE_RESP_VERSION_1	1
+	__u32	version;
+#define IOMMU_PAGE_RESP_PASID_VALID	(1 << 0)
+	__u32	flags;
+	__u32	pasid;
+	__u32	grpid;
+	__u32	code;
+};
+
+
 /* iommu fault flags */
 #define IOMMU_FAULT_READ	0x0
 #define IOMMU_FAULT_WRITE	0x1
include/uapi/linux/iommu.h deleted +0 -161

161 lines not on this page · raw message

MAINTAINERS +0 -1
index 90f13281d297..e380bb987049 100644
@@ -10984,7 +10984,6 @@ F:	drivers/iommu/
 F:	include/linux/iommu.h
 F:	include/linux/iova.h
 F:	include/linux/of_iommu.h
-F:	include/uapi/linux/iommu.h
 
 IOMMUFD
 M:	Jason Gunthorpe <[email protected]>
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

PATCH v5 2/12 iommu/arm-smmu-v3: Remove unrecoverable faults reporting

In reply to Lu Baolu

No device driver registers fault handler to handle the reported unrecoveraable faults. Remove it to avoid dead code.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 46 ++++++---------------
 1 file changed, 13 insertions(+), 33 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +13 -33
index e82bf1c449a3..90b281eda73f 100644
@@ -1469,7 +1469,6 @@ arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)
 static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
 {
 	int ret;
-	u32 reason;
 	u32 perm = 0;
 	struct arm_smmu_master *master;
 	bool ssid_valid = evt[0] & EVTQ_0_SSV;
@@ -1479,16 +1478,9 @@ static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
 
 	switch (FIELD_GET(EVTQ_0_ID, evt[0])) {
 	case EVT_ID_TRANSLATION_FAULT:
-		reason = IOMMU_FAULT_REASON_PTE_FETCH;
-		break;
 	case EVT_ID_ADDR_SIZE_FAULT:
-		reason = IOMMU_FAULT_REASON_OOR_ADDRESS;
-		break;
 	case EVT_ID_ACCESS_FAULT:
-		reason = IOMMU_FAULT_REASON_ACCESS;
-		break;
 	case EVT_ID_PERMISSION_FAULT:
-		reason = IOMMU_FAULT_REASON_PERMISSION;
 		break;
 	default:
 		return -EOPNOTSUPP;
@@ -1498,6 +1490,9 @@ static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
 	if (evt[1] & EVTQ_1_S2)
 		return -EFAULT;
 
+	if (!(evt[1] & EVTQ_1_STALL))
+		return -EOPNOTSUPP;
+
 	if (evt[1] & EVTQ_1_RnW)
 		perm |= IOMMU_FAULT_PERM_READ;
 	else
@@ -1509,32 +1504,17 @@ static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
 	if (evt[1] & EVTQ_1_PnU)
 		perm |= IOMMU_FAULT_PERM_PRIV;
 
-	if (evt[1] & EVTQ_1_STALL) {
-		flt->type = IOMMU_FAULT_PAGE_REQ;
-		flt->prm = (struct iommu_fault_page_request) {
-			.flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,
-			.grpid = FIELD_GET(EVTQ_1_STAG, evt[1]),
-			.perm = perm,
-			.addr = FIELD_GET(EVTQ_2_ADDR, evt[2]),
-		};
+	flt->type = IOMMU_FAULT_PAGE_REQ;
+	flt->prm = (struct iommu_fault_page_request) {
+		.flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,
+		.grpid = FIELD_GET(EVTQ_1_STAG, evt[1]),
+		.perm = perm,
+		.addr = FIELD_GET(EVTQ_2_ADDR, evt[2]),
+	};
 
-		if (ssid_valid) {
-			flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
-			flt->prm.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]);
-		}
-	} else {
-		flt->type = IOMMU_FAULT_DMA_UNRECOV;
-		flt->event = (struct iommu_fault_unrecoverable) {
-			.reason = reason,
-			.flags = IOMMU_FAULT_UNRECOV_ADDR_VALID,
-			.perm = perm,
-			.addr = FIELD_GET(EVTQ_2_ADDR, evt[2]),
-		};
-
-		if (ssid_valid) {
-			flt->event.flags |= IOMMU_FAULT_UNRECOV_PASID_VALID;
-			flt->event.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]);
-		}
+	if (ssid_valid) {
+		flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
+		flt->prm.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]);
 	}
 
 	mutex_lock(&smmu->streams_mutex);
Signature · 1 line
2.34.1
author maintainer
#3 lore raw

PATCH v5 3/12 iommu: Remove unrecoverable fault data

In reply to Lu Baolu

The unrecoverable fault data is not used anywhere. Remove it to avoid dead code.

---
 include/linux/iommu.h | 70 +------------------------------------------
 1 file changed, 1 insertion(+), 69 deletions(-)
include/linux/iommu.h +1 -69
index 895405567f86..1040047982aa 100644
@@ -48,69 +48,9 @@ struct iommu_dma_cookie;
 
 /* Generic fault types, can be expanded IRQ remapping fault */
 enum iommu_fault_type {
-	IOMMU_FAULT_DMA_UNRECOV = 1,	/* unrecoverable fault */
 	IOMMU_FAULT_PAGE_REQ,		/* page request fault */
 };
 
-enum iommu_fault_reason {
-	IOMMU_FAULT_REASON_UNKNOWN = 0,
-
-	/* Could not access the PASID table (fetch caused external abort) */
-	IOMMU_FAULT_REASON_PASID_FETCH,
-
-	/* PASID entry is invalid or has configuration errors */
-	IOMMU_FAULT_REASON_BAD_PASID_ENTRY,
-
-	/*
-	 * PASID is out of range (e.g. exceeds the maximum PASID
-	 * supported by the IOMMU) or disabled.
-	 */
-	IOMMU_FAULT_REASON_PASID_INVALID,
-
-	/*
-	 * An external abort occurred fetching (or updating) a translation
-	 * table descriptor
-	 */
-	IOMMU_FAULT_REASON_WALK_EABT,
-
-	/*
-	 * Could not access the page table entry (Bad address),
-	 * actual translation fault
-	 */
-	IOMMU_FAULT_REASON_PTE_FETCH,
-
-	/* Protection flag check failed */
-	IOMMU_FAULT_REASON_PERMISSION,
-
-	/* access flag check failed */
-	IOMMU_FAULT_REASON_ACCESS,
-
-	/* Output address of a translation stage caused Address Size fault */
-	IOMMU_FAULT_REASON_OOR_ADDRESS,
-};
-
-/**
- * struct iommu_fault_unrecoverable - Unrecoverable fault data
- * @reason: reason of the fault, from &enum iommu_fault_reason
- * @flags: parameters of this fault (IOMMU_FAULT_UNRECOV_* values)
- * @pasid: Process Address Space ID
- * @perm: requested permission access using by the incoming transaction
- *        (IOMMU_FAULT_PERM_* values)
- * @addr: offending page address
- * @fetch_addr: address that caused a fetch abort, if any
- */
-struct iommu_fault_unrecoverable {
-	__u32	reason;
-#define IOMMU_FAULT_UNRECOV_PASID_VALID		(1 << 0)
-#define IOMMU_FAULT_UNRECOV_ADDR_VALID		(1 << 1)
-#define IOMMU_FAULT_UNRECOV_FETCH_ADDR_VALID	(1 << 2)
-	__u32	flags;
-	__u32	pasid;
-	__u32	perm;
-	__u64	addr;
-	__u64	fetch_addr;
-};
-
 /**
  * struct iommu_fault_page_request - Page Request data
  * @flags: encodes whether the corresponding fields are valid and whether this
@@ -140,19 +80,11 @@ struct iommu_fault_page_request {
 /**
  * struct iommu_fault - Generic fault data
  * @type: fault type from &enum iommu_fault_type
- * @padding: reserved for future use (should be zero)
- * @event: fault event, when @type is %IOMMU_FAULT_DMA_UNRECOV
  * @prm: Page Request message, when @type is %IOMMU_FAULT_PAGE_REQ
- * @padding2: sets the fault size to allow for future extensions
  */
 struct iommu_fault {
 	__u32	type;
-	__u32	padding;
-	union {
-		struct iommu_fault_unrecoverable event;
-		struct iommu_fault_page_request prm;
-		__u8 padding2[56];
-	};
+	struct iommu_fault_page_request prm;
 };
 
 /**
Signature · 1 line
2.34.1
author maintainer
#4 lore raw

PATCH v5 4/12 iommu: Cleanup iopf data structure definitions

In reply to Lu Baolu

struct iommu_fault_page_request and struct iommu_page_response are not part of uAPI anymore. Convert them to data structures for kAPI.

---
 include/linux/iommu.h      | 27 +++++++++++----------------
 drivers/iommu/io-pgfault.c |  1 -
 drivers/iommu/iommu.c      |  4 ----
 3 files changed, 11 insertions(+), 21 deletions(-)
include/linux/iommu.h +11 -16
index 1040047982aa..db13ebaabcf7 100644
@@ -69,12 +69,12 @@ struct iommu_fault_page_request {
 #define IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE	(1 << 1)
 #define IOMMU_FAULT_PAGE_REQUEST_PRIV_DATA	(1 << 2)
 #define IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID	(1 << 3)
-	__u32	flags;
-	__u32	pasid;
-	__u32	grpid;
-	__u32	perm;
-	__u64	addr;
-	__u64	private_data[2];
+	u32	flags;
+	u32	pasid;
+	u32	grpid;
+	u32	perm;
+	u64	addr;
+	u64	private_data[2];
 };
 
 /**
@@ -83,7 +83,7 @@ struct iommu_fault_page_request {
  * @prm: Page Request message, when @type is %IOMMU_FAULT_PAGE_REQ
  */
 struct iommu_fault {
-	__u32	type;
+	u32 type;
 	struct iommu_fault_page_request prm;
 };
 
@@ -104,8 +104,6 @@ enum iommu_page_response_code {
 
 /**
  * struct iommu_page_response - Generic page response information
- * @argsz: User filled size of this data
- * @version: API version of this structure
  * @flags: encodes whether the corresponding fields are valid
  *         (IOMMU_FAULT_PAGE_RESPONSE_* values)
  * @pasid: Process Address Space ID
@@ -113,14 +111,11 @@ enum iommu_page_response_code {
  * @code: response code from &enum iommu_page_response_code
  */
 struct iommu_page_response {
-	__u32	argsz;
-#define IOMMU_PAGE_RESP_VERSION_1	1
-	__u32	version;
 #define IOMMU_PAGE_RESP_PASID_VALID	(1 << 0)
-	__u32	flags;
-	__u32	pasid;
-	__u32	grpid;
-	__u32	code;
+	u32	flags;
+	u32	pasid;
+	u32	grpid;
+	u32	code;
 };
 
 
drivers/iommu/io-pgfault.c +0 -1
index e5b8b9110c13..24b5545352ae 100644
@@ -56,7 +56,6 @@ static int iopf_complete_group(struct device *dev, struct iopf_fault *iopf,
 			       enum iommu_page_response_code status)
 {
 	struct iommu_page_response resp = {
-		.version		= IOMMU_PAGE_RESP_VERSION_1,
 		.pasid			= iopf->fault.prm.pasid,
 		.grpid			= iopf->fault.prm.grpid,
 		.code			= status,
drivers/iommu/iommu.c +0 -4
index 3bfc56df4f78..ccad2131f49d 100644
@@ -1458,10 +1458,6 @@ int iommu_page_response(struct device *dev,
 	if (!param || !param->fault_param)
 		return -EINVAL;
 
-	if (msg->version != IOMMU_PAGE_RESP_VERSION_1 ||
-	    msg->flags & ~IOMMU_PAGE_RESP_PASID_VALID)
-		return -EINVAL;
-
 	/* Only send response if there is a fault report pending */
 	mutex_lock(&param->fault_param->lock);
 	if (list_empty(&param->fault_param->faults)) {
Signature · 1 line
2.34.1
author maintainer
#5 lore raw

PATCH v5 5/12 iommu: Merge iopf_device_param into iommu_fault_param

In reply to Lu Baolu

The struct dev_iommu contains two pointers, fault_param and iopf_param. The fault_param pointer points to a data structure that is used to store pending faults that are awaiting responses. The iopf_param pointer points to a data structure that is used to store partial faults that are part of a Page Request Group.

The fault_param and iopf_param pointers are essentially duplicate. This causes memory waste. Merge the iopf_device_param pointer into the iommu_fault_param pointer to consolidate the code and save memory. The consolidated pointer would be allocated on demand when the device driver enables the iopf on device, and would be freed after iopf is disabled.

---
 include/linux/iommu.h      |  18 ++++--
 drivers/iommu/io-pgfault.c | 113 ++++++++++++++++++-------------------
 drivers/iommu/iommu.c      |  34 ++---------
 3 files changed, 75 insertions(+), 90 deletions(-)
include/linux/iommu.h +14 -4
index db13ebaabcf7..7bb2efbb1b34 100644
@@ -40,6 +40,7 @@ struct notifier_block;
 struct iommu_sva;
 struct iommu_fault_event;
 struct iommu_dma_cookie;
+struct iopf_queue;
 
 #define IOMMU_FAULT_PERM_READ	(1 << 0) /* read */
 #define IOMMU_FAULT_PERM_WRITE	(1 << 1) /* write */
@@ -473,21 +474,31 @@ struct iommu_fault_event {
  * struct iommu_fault_param - per-device IOMMU fault data
  * @handler: Callback function to handle IOMMU faults at device level
  * @data: handler private data
- * @faults: holds the pending faults which needs response
  * @lock: protect pending faults list
+ * @dev: the device that owns this param
+ * @queue: IOPF queue
+ * @queue_list: index into queue->devices
+ * @partial: faults that are part of a Page Request Group for which the last
+ *           request hasn't been submitted yet.
+ * @faults: holds the pending faults which needs response
  */
 struct iommu_fault_param {
 	iommu_dev_fault_handler_t handler;
 	void *data;
+	struct mutex lock;
+
+	struct device *dev;
+	struct iopf_queue *queue;
+	struct list_head queue_list;
+
+	struct list_head partial;
 	struct list_head faults;
-	struct mutex lock;
 };
 
 /**
  * struct dev_iommu - Collection of per-device IOMMU data
  *
  * @fault_param: IOMMU detected device fault reporting data
- * @iopf_param:	 I/O Page Fault queue and data
  * @fwspec:	 IOMMU fwspec data
  * @iommu_dev:	 IOMMU device this device is linked to
  * @priv:	 IOMMU Driver private data
@@ -502,7 +513,6 @@ struct iommu_fault_param {
 struct dev_iommu {
 	struct mutex lock;
 	struct iommu_fault_param	*fault_param;
-	struct iopf_device_param	*iopf_param;
 	struct iommu_fwspec		*fwspec;
 	struct iommu_device		*iommu_dev;
 	void				*priv;
drivers/iommu/io-pgfault.c +55 -58

186 lines not on this page · raw message

drivers/iommu/iommu.c +6 -28
index ccad2131f49d..7a8c9ba0401d 100644
@@ -1319,27 +1319,18 @@ int iommu_register_device_fault_handler(struct device *dev,
 	struct dev_iommu *param = dev->iommu;
 	int ret = 0;
 
-	if (!param)
+	if (!param || !param->fault_param)
 		return -EINVAL;
 
 	mutex_lock(&param->lock);
 	/* Only allow one fault handler registered for each device */
-	if (param->fault_param) {
+	if (param->fault_param->handler) {
 		ret = -EBUSY;
 		goto done_unlock;
 	}
 
-	get_device(dev);
-	param->fault_param = kzalloc(sizeof(*param->fault_param), GFP_KERNEL);
-	if (!param->fault_param) {
-		put_device(dev);
-		ret = -ENOMEM;
-		goto done_unlock;
-	}
 	param->fault_param->handler = handler;
 	param->fault_param->data = data;
-	mutex_init(&param->fault_param->lock);
-	INIT_LIST_HEAD(&param->fault_param->faults);
 
 done_unlock:
 	mutex_unlock(&param->lock);
@@ -1360,29 +1351,16 @@ EXPORT_SYMBOL_GPL(iommu_register_device_fault_handler);
 int iommu_unregister_device_fault_handler(struct device *dev)
 {
 	struct dev_iommu *param = dev->iommu;
-	int ret = 0;
 
-	if (!param)
+	if (!param || !param->fault_param)
 		return -EINVAL;
 
 	mutex_lock(&param->lock);
-
-	if (!param->fault_param)
-		goto unlock;
-
-	/* we cannot unregister handler if there are pending faults */
-	if (!list_empty(&param->fault_param->faults)) {
-		ret = -EBUSY;
-		goto unlock;
-	}
-
-	kfree(param->fault_param);
-	param->fault_param = NULL;
-	put_device(dev);
-unlock:
+	param->fault_param->handler = NULL;
+	param->fault_param->data = NULL;
 	mutex_unlock(&param->lock);
 
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(iommu_unregister_device_fault_handler);
 
Signature · 1 line
2.34.1
author maintainer
#6 lore raw

PATCH v5 6/12 iommu: Remove iommu_[un]register_device_fault_handler()

In reply to Lu Baolu

The individual iommu driver reports the iommu page faults by calling iommu_report_device_fault(), where a pre-registered device fault handler is called to route the fault to another fault handler installed on the corresponding iommu domain.

The pre-registered device fault handler is static and won't be dynamic as the fault handler is eventually per iommu domain. Replace calling device fault handler with iommu_queue_iopf().

After this replacement, the registering and unregistering fault handler interfaces are not needed anywhere. Remove the interfaces and the related data structures to avoid dead code.

Convert cookie parameter of iommu_queue_iopf() into a device pointer that is really passed.

---
 include/linux/iommu.h                         | 23 ------
 drivers/iommu/iommu-sva.h                     |  4 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   | 13 +---
 drivers/iommu/intel/iommu.c                   | 24 ++----
 drivers/iommu/io-pgfault.c                    |  6 +-
 drivers/iommu/iommu.c                         | 76 +------------------
 6 files changed, 13 insertions(+), 133 deletions(-)
include/linux/iommu.h +0 -23
index 7bb2efbb1b34..947f074d9e8e 100644
@@ -126,7 +126,6 @@ struct iommu_page_response {
 
 typedef int (*iommu_fault_handler_t)(struct iommu_domain *,
 			struct device *, unsigned long, int, void *);
-typedef int (*iommu_dev_fault_handler_t)(struct iommu_fault *, void *);
 
 struct iommu_domain_geometry {
 	dma_addr_t aperture_start; /* First address that can be mapped    */
@@ -472,8 +471,6 @@ struct iommu_fault_event {
 
 /**
  * struct iommu_fault_param - per-device IOMMU fault data
- * @handler: Callback function to handle IOMMU faults at device level
- * @data: handler private data
  * @lock: protect pending faults list
  * @dev: the device that owns this param
  * @queue: IOPF queue
@@ -483,8 +480,6 @@ struct iommu_fault_event {
  * @faults: holds the pending faults which needs response
  */
 struct iommu_fault_param {
-	iommu_dev_fault_handler_t handler;
-	void *data;
 	struct mutex lock;
 
 	struct device *dev;
@@ -605,11 +600,6 @@ extern int iommu_group_for_each_dev(struct iommu_group *group, void *data,
 extern struct iommu_group *iommu_group_get(struct device *dev);
 extern struct iommu_group *iommu_group_ref_get(struct iommu_group *group);
 extern void iommu_group_put(struct iommu_group *group);
-extern int iommu_register_device_fault_handler(struct device *dev,
-					iommu_dev_fault_handler_t handler,
-					void *data);
-
-extern int iommu_unregister_device_fault_handler(struct device *dev);
 
 extern int iommu_report_device_fault(struct device *dev,
 				     struct iommu_fault_event *evt);
@@ -993,19 +983,6 @@ static inline void iommu_group_put(struct iommu_group *group)
 {
 }
 
-static inline
-int iommu_register_device_fault_handler(struct device *dev,
-					iommu_dev_fault_handler_t handler,
-					void *data)
-{
-	return -ENODEV;
-}
-
-static inline int iommu_unregister_device_fault_handler(struct device *dev)
-{
-	return 0;
-}
-
 static inline
 int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt)
 {
drivers/iommu/iommu-sva.h +2 -2
index 54946b5a7caf..de7819c796ce 100644
@@ -13,7 +13,7 @@ struct iommu_fault;
 struct iopf_queue;
 
 #ifdef CONFIG_IOMMU_SVA
-int iommu_queue_iopf(struct iommu_fault *fault, void *cookie);
+int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev);
 
 int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev);
 int iopf_queue_remove_device(struct iopf_queue *queue,
@@ -26,7 +26,7 @@ enum iommu_page_response_code
 iommu_sva_handle_iopf(struct iommu_fault *fault, void *data);
 
 #else /* CONFIG_IOMMU_SVA */
-static inline int iommu_queue_iopf(struct iommu_fault *fault, void *cookie)
+static inline int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
 {
 	return -ENODEV;
 }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -12
index 4d83edc2be99..4238c97ba07a 100644
@@ -448,7 +448,6 @@ bool arm_smmu_master_sva_enabled(struct arm_smmu_master *master)
 
 static int arm_smmu_master_sva_enable_iopf(struct arm_smmu_master *master)
 {
-	int ret;
 	struct device *dev = master->dev;
 
 	/*
@@ -461,16 +460,7 @@ static int arm_smmu_master_sva_enable_iopf(struct arm_smmu_master *master)
 	if (!master->iopf_enabled)
 		return -EINVAL;
 
-	ret = iopf_queue_add_device(master->smmu->evtq.iopf, dev);
-	if (ret)
-		return ret;
-
-	ret = iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev);
-	if (ret) {
-		iopf_queue_remove_device(master->smmu->evtq.iopf, dev);
-		return ret;
-	}
-	return 0;
+	return iopf_queue_add_device(master->smmu->evtq.iopf, dev);
 }
 
 static void arm_smmu_master_sva_disable_iopf(struct arm_smmu_master *master)
@@ -480,7 +470,6 @@ static void arm_smmu_master_sva_disable_iopf(struct arm_smmu_master *master)
 	if (!master->iopf_enabled)
 		return;
 
-	iommu_unregister_device_fault_handler(dev);
 	iopf_queue_remove_device(master->smmu->evtq.iopf, dev);
 }
 
drivers/iommu/intel/iommu.c +7 -17
index 5db283c17e0d..7b0ed6cdd6b9 100644
@@ -4575,23 +4575,15 @@ static int intel_iommu_enable_iopf(struct device *dev)
 	if (ret)
 		return ret;
 
-	ret = iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev);
-	if (ret)
-		goto iopf_remove_device;
-
 	ret = pci_enable_pri(pdev, PRQ_DEPTH);
-	if (ret)
-		goto iopf_unregister_handler;
+	if (ret) {
+		iopf_queue_remove_device(iommu->iopf_queue, dev);
+		return ret;
+	}
+
 	info->pri_enabled = 1;
 
 	return 0;
-
-iopf_unregister_handler:
-	iommu_unregister_device_fault_handler(dev);
-iopf_remove_device:
-	iopf_queue_remove_device(iommu->iopf_queue, dev);
-
-	return ret;
 }
 
 static int intel_iommu_disable_iopf(struct device *dev)
@@ -4614,11 +4606,9 @@ static int intel_iommu_disable_iopf(struct device *dev)
 	info->pri_enabled = 0;
 
 	/*
-	 * With PRI disabled and outstanding PRQs drained, unregistering
-	 * fault handler and removing device from iopf queue should never
-	 * fail.
+	 * With PRI disabled and outstanding PRQs drained, removing device
+	 * from iopf queue should never fail.
 	 */
-	WARN_ON(iommu_unregister_device_fault_handler(dev));
 	WARN_ON(iopf_queue_remove_device(iommu->iopf_queue, dev));
 
 	return 0;
drivers/iommu/io-pgfault.c +2 -4
index b1cf28055525..31832aeacdba 100644
@@ -87,7 +87,7 @@ static void iopf_handler(struct work_struct *work)
 /**
  * iommu_queue_iopf - IO Page Fault handler
  * @fault: fault event
- * @cookie: struct device, passed to iommu_register_device_fault_handler.
+ * @dev: struct device.
  *
  * Add a fault to the device workqueue, to be handled by mm.
  *
@@ -124,14 +124,12 @@ static void iopf_handler(struct work_struct *work)
  *
  * Return: 0 on success and <0 on error.
  */
-int iommu_queue_iopf(struct iommu_fault *fault, void *cookie)
+int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
 {
 	int ret;
 	struct iopf_group *group;
 	struct iopf_fault *iopf, *next;
 	struct iommu_fault_param *iopf_param;
-
-	struct device *dev = cookie;
 	struct dev_iommu *param = dev->iommu;
 
 	lockdep_assert_held(&param->lock);
drivers/iommu/iommu.c +1 -75

94 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#7 lore raw

PATCH v5 7/12 iommu: Merge iommu_fault_event and iopf_fault

In reply to Lu Baolu

The iommu_fault_event and iopf_fault data structures store the same information about an iopf fault. They are also used in the same way. Merge these two data structures into a single one to make the code more concise and easier to maintain.

---
 include/linux/iommu.h                       | 27 ++++++---------------
 drivers/iommu/intel/iommu.h                 |  2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  4 +--
 drivers/iommu/intel/svm.c                   |  5 ++--
 drivers/iommu/io-pgfault.c                  |  5 ----
 drivers/iommu/iommu.c                       |  8 +++---
 6 files changed, 17 insertions(+), 34 deletions(-)
include/linux/iommu.h +8 -19
index 947f074d9e8e..f2f32e729d2c 100644
@@ -38,7 +38,6 @@ struct iommu_domain;
 struct iommu_domain_ops;
 struct notifier_block;
 struct iommu_sva;
-struct iommu_fault_event;
 struct iommu_dma_cookie;
 struct iopf_queue;
 
@@ -119,6 +118,11 @@ struct iommu_page_response {
 	u32	code;
 };
 
+struct iopf_fault {
+	struct iommu_fault fault;
+	/* node for pending lists */
+	struct list_head list;
+};
 
 /* iommu fault flags */
 #define IOMMU_FAULT_READ	0x0
@@ -362,7 +366,7 @@ struct iommu_ops {
 	int (*dev_disable_feat)(struct device *dev, enum iommu_dev_features f);
 
 	int (*page_response)(struct device *dev,
-			     struct iommu_fault_event *evt,
+			     struct iopf_fault *evt,
 			     struct iommu_page_response *msg);
 
 	int (*def_domain_type)(struct device *dev);
@@ -455,20 +459,6 @@ struct iommu_device {
 	u32 max_pasids;
 };
 
-/**
- * struct iommu_fault_event - Generic fault event
- *
- * Can represent recoverable faults such as a page requests or
- * unrecoverable faults such as DMA or IRQ remapping faults.
- *
- * @fault: fault descriptor
- * @list: pending fault event list, used for tracking responses
- */
-struct iommu_fault_event {
-	struct iommu_fault fault;
-	struct list_head list;
-};
-
 /**
  * struct iommu_fault_param - per-device IOMMU fault data
  * @lock: protect pending faults list
@@ -601,8 +591,7 @@ extern struct iommu_group *iommu_group_get(struct device *dev);
 extern struct iommu_group *iommu_group_ref_get(struct iommu_group *group);
 extern void iommu_group_put(struct iommu_group *group);
 
-extern int iommu_report_device_fault(struct device *dev,
-				     struct iommu_fault_event *evt);
+extern int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt);
 extern int iommu_page_response(struct device *dev,
 			       struct iommu_page_response *msg);
 
@@ -984,7 +973,7 @@ static inline void iommu_group_put(struct iommu_group *group)
 }
 
 static inline
-int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt)
+int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt)
 {
 	return -ENODEV;
 }
drivers/iommu/intel/iommu.h +1 -1
index c18fb699c87a..66221bb206f4 100644
@@ -847,7 +847,7 @@ struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn);
 void intel_svm_check(struct intel_iommu *iommu);
 int intel_svm_enable_prq(struct intel_iommu *iommu);
 int intel_svm_finish_prq(struct intel_iommu *iommu);
-int intel_svm_page_response(struct device *dev, struct iommu_fault_event *evt,
+int intel_svm_page_response(struct device *dev, struct iopf_fault *evt,
 			    struct iommu_page_response *msg);
 struct iommu_domain *intel_svm_domain_alloc(void);
 void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t pasid);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -2
index 90b281eda73f..e537b6c046c5 100644
@@ -922,7 +922,7 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 }
 
 static int arm_smmu_page_response(struct device *dev,
-				  struct iommu_fault_event *unused,
+				  struct iopf_fault *unused,
 				  struct iommu_page_response *resp)
 {
 	struct arm_smmu_cmdq_ent cmd = {0};
@@ -1473,7 +1473,7 @@ static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
 	struct arm_smmu_master *master;
 	bool ssid_valid = evt[0] & EVTQ_0_SSV;
 	u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]);
-	struct iommu_fault_event fault_evt = { };
+	struct iopf_fault fault_evt = { };
 	struct iommu_fault *flt = &fault_evt.fault;
 
 	switch (FIELD_GET(EVTQ_0_ID, evt[0])) {
drivers/iommu/intel/svm.c +2 -3
index 50a481c895b8..9de349ea215c 100644
@@ -543,13 +543,12 @@ static int prq_to_iommu_prot(struct page_req_dsc *req)
 static int intel_svm_prq_report(struct intel_iommu *iommu, struct device *dev,
 				struct page_req_dsc *desc)
 {
-	struct iommu_fault_event event;
+	struct iopf_fault event = { };
 
 	if (!dev || !dev_is_pci(dev))
 		return -ENODEV;
 
 	/* Fill in event data for device specific processing */
-	memset(&event, 0, sizeof(struct iommu_fault_event));
 	event.fault.type = IOMMU_FAULT_PAGE_REQ;
 	event.fault.prm.addr = (u64)desc->addr << VTD_PAGE_SHIFT;
 	event.fault.prm.pasid = desc->pasid;
@@ -721,7 +720,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 }
 
 int intel_svm_page_response(struct device *dev,
-			    struct iommu_fault_event *evt,
+			    struct iopf_fault *evt,
 			    struct iommu_page_response *msg)
 {
 	struct iommu_fault_page_request *prm;
drivers/iommu/io-pgfault.c +0 -5
index 31832aeacdba..c45977bb7da3 100644
@@ -25,11 +25,6 @@ struct iopf_queue {
 	struct mutex			lock;
 };
 
-struct iopf_fault {
-	struct iommu_fault		fault;
-	struct list_head		list;
-};
-
 struct iopf_group {
 	struct iopf_fault		last_fault;
 	struct list_head		faults;
drivers/iommu/iommu.c +4 -4
index bdc0d14ef29f..22600a5e8262 100644
@@ -1305,10 +1305,10 @@ EXPORT_SYMBOL_GPL(iommu_group_put);
  *
  * Return 0 on success, or an error.
  */
-int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt)
+int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt)
 {
 	struct dev_iommu *param = dev->iommu;
-	struct iommu_fault_event *evt_pending = NULL;
+	struct iopf_fault *evt_pending = NULL;
 	struct iommu_fault_param *fparam;
 	int ret = 0;
 
@@ -1321,7 +1321,7 @@ int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt)
 
 	if (evt->fault.type == IOMMU_FAULT_PAGE_REQ &&
 	    (evt->fault.prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE)) {
-		evt_pending = kmemdup(evt, sizeof(struct iommu_fault_event),
+		evt_pending = kmemdup(evt, sizeof(struct iopf_fault),
 				      GFP_KERNEL);
 		if (!evt_pending) {
 			ret = -ENOMEM;
@@ -1350,7 +1350,7 @@ int iommu_page_response(struct device *dev,
 {
 	bool needs_pasid;
 	int ret = -EINVAL;
-	struct iommu_fault_event *evt;
+	struct iopf_fault *evt;
 	struct iommu_fault_page_request *prm;
 	struct dev_iommu *param = dev->iommu;
 	const struct iommu_ops *ops = dev_iommu_ops(dev);
Signature · 1 line
2.34.1
Reviewed-by on v5 7/12
#8 lore raw

RE: [PATCH v5 07/12] iommu: Merge iommu_fault_event and iopf_fault

Quoted 9 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

The iommu_fault_event and iopf_fault data structures store the same information about an iopf fault. They are also used in the same way. Merge these two data structures into a single one to make the code more concise and easier to maintain.

author maintainer
#9 lore raw

PATCH v5 8/12 iommu: Prepare for separating SVA and IOPF

In reply to Lu Baolu

Move iopf_group data structure to iommu.h to make it a minimal set of faults that a domain's page fault handler should handle.

Add a new function, iopf_free_group(), to free a fault group after all faults in the group are handled. This function will be made global so that it can be called from other files, such as iommu-sva.c.

Move iopf_queue data structure to iommu.h to allow the workqueue to be scheduled out of this file.

This will simplify the sequential patches.

---
 include/linux/iommu.h      | 20 +++++++++++++++++++-
 drivers/iommu/io-pgfault.c | 37 +++++++++++++------------------------
 2 files changed, 32 insertions(+), 25 deletions(-)
include/linux/iommu.h +19 -1
index f2f32e729d2c..b179594fc378 100644
@@ -39,7 +39,6 @@ struct iommu_domain_ops;
 struct notifier_block;
 struct iommu_sva;
 struct iommu_dma_cookie;
-struct iopf_queue;
 
 #define IOMMU_FAULT_PERM_READ	(1 << 0) /* read */
 #define IOMMU_FAULT_PERM_WRITE	(1 << 1) /* write */
@@ -124,6 +123,25 @@ struct iopf_fault {
 	struct list_head list;
 };
 
+struct iopf_group {
+	struct iopf_fault last_fault;
+	struct list_head faults;
+	struct work_struct work;
+	struct device *dev;
+};
+
+/**
+ * struct iopf_queue - IO Page Fault queue
+ * @wq: the fault workqueue
+ * @devices: devices attached to this queue
+ * @lock: protects the device list
+ */
+struct iopf_queue {
+	struct workqueue_struct *wq;
+	struct list_head devices;
+	struct mutex lock;
+};
+
 /* iommu fault flags */
 #define IOMMU_FAULT_READ	0x0
 #define IOMMU_FAULT_WRITE	0x1
drivers/iommu/io-pgfault.c +13 -24
index c45977bb7da3..09e05f483b4f 100644
@@ -13,24 +13,17 @@
 
 #include "iommu-sva.h"
 
-/**
- * struct iopf_queue - IO Page Fault queue
- * @wq: the fault workqueue
- * @devices: devices attached to this queue
- * @lock: protects the device list
- */
-struct iopf_queue {
-	struct workqueue_struct		*wq;
-	struct list_head		devices;
-	struct mutex			lock;
-};
+static void iopf_free_group(struct iopf_group *group)
+{
+	struct iopf_fault *iopf, *next;
 
-struct iopf_group {
-	struct iopf_fault		last_fault;
-	struct list_head		faults;
-	struct work_struct		work;
-	struct device			*dev;
-};
+	list_for_each_entry_safe(iopf, next, &group->faults, list) {
+		if (!(iopf->fault.prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE))
+			kfree(iopf);
+	}
+
+	kfree(group);
+}
 
 static int iopf_complete_group(struct device *dev, struct iopf_fault *iopf,
 			       enum iommu_page_response_code status)
@@ -50,9 +43,9 @@ static int iopf_complete_group(struct device *dev, struct iopf_fault *iopf,
 
 static void iopf_handler(struct work_struct *work)
 {
+	struct iopf_fault *iopf;
 	struct iopf_group *group;
 	struct iommu_domain *domain;
-	struct iopf_fault *iopf, *next;
 	enum iommu_page_response_code status = IOMMU_PAGE_RESP_SUCCESS;
 
 	group = container_of(work, struct iopf_group, work);
@@ -61,7 +54,7 @@ static void iopf_handler(struct work_struct *work)
 	if (!domain || !domain->iopf_handler)
 		status = IOMMU_PAGE_RESP_INVALID;
 
-	list_for_each_entry_safe(iopf, next, &group->faults, list) {
+	list_for_each_entry(iopf, &group->faults, list) {
 		/*
 		 * For the moment, errors are sticky: don't handle subsequent
 		 * faults in the group if there is an error.
@@ -69,14 +62,10 @@ static void iopf_handler(struct work_struct *work)
 		if (status == IOMMU_PAGE_RESP_SUCCESS)
 			status = domain->iopf_handler(&iopf->fault,
 						      domain->fault_data);
-
-		if (!(iopf->fault.prm.flags &
-		      IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE))
-			kfree(iopf);
 	}
 
 	iopf_complete_group(group->dev, &group->last_fault, status);
-	kfree(group);
+	iopf_free_group(group);
 }
 
 /**
Signature · 1 line
2.34.1
author maintainer
#10 lore raw

PATCH v5 9/12 iommu: Make iommu_queue_iopf() more generic

In reply to Lu Baolu

Make iommu_queue_iopf() more generic by making the iopf_group a minimal set of iopf's that an iopf handler of domain should handle and respond to. Add domain parameter to struct iopf_group so that the handler can retrieve and use it directly.

Change iommu_queue_iopf() to forward groups of iopf's to the domain's iopf handler. This is also a necessary step to decouple the sva iopf handling code from this interface.

---
 include/linux/iommu.h      |  4 ++--
 drivers/iommu/iommu-sva.h  |  6 ++---
 drivers/iommu/io-pgfault.c | 49 ++++++++++++++++++++++++++++----------
 drivers/iommu/iommu-sva.c  |  3 +--
 4 files changed, 42 insertions(+), 20 deletions(-)
include/linux/iommu.h +2 -2
index b179594fc378..da47813c2e4c 100644
@@ -128,6 +128,7 @@ struct iopf_group {
 	struct list_head faults;
 	struct work_struct work;
 	struct device *dev;
+	struct iommu_domain *domain;
 };
 
 /**
@@ -197,8 +198,7 @@ struct iommu_domain {
 	unsigned long pgsize_bitmap;	/* Bitmap of page sizes in use */
 	struct iommu_domain_geometry geometry;
 	struct iommu_dma_cookie *iova_cookie;
-	enum iommu_page_response_code (*iopf_handler)(struct iommu_fault *fault,
-						      void *data);
+	int (*iopf_handler)(struct iopf_group *group);
 	void *fault_data;
 	union {
 		struct {
drivers/iommu/iommu-sva.h +2 -4
index de7819c796ce..27c8da115b41 100644
@@ -22,8 +22,7 @@ int iopf_queue_flush_dev(struct device *dev);
 struct iopf_queue *iopf_queue_alloc(const char *name);
 void iopf_queue_free(struct iopf_queue *queue);
 int iopf_queue_discard_partial(struct iopf_queue *queue);
-enum iommu_page_response_code
-iommu_sva_handle_iopf(struct iommu_fault *fault, void *data);
+int iommu_sva_handle_iopf(struct iopf_group *group);
 
 #else /* CONFIG_IOMMU_SVA */
 static inline int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
@@ -62,8 +61,7 @@ static inline int iopf_queue_discard_partial(struct iopf_queue *queue)
 	return -ENODEV;
 }
 
-static inline enum iommu_page_response_code
-iommu_sva_handle_iopf(struct iommu_fault *fault, void *data)
+static inline int iommu_sva_handle_iopf(struct iopf_group *group)
 {
 	return IOMMU_PAGE_RESP_INVALID;
 }
drivers/iommu/io-pgfault.c +37 -12

101 lines not on this page · raw message

drivers/iommu/iommu-sva.c +1 -2
index b78671a8a914..ba0d5b7e106a 100644
@@ -149,11 +149,10 @@ EXPORT_SYMBOL_GPL(iommu_sva_get_pasid);
  * I/O page fault handler for SVA
  */
 enum iommu_page_response_code
-iommu_sva_handle_iopf(struct iommu_fault *fault, void *data)
+iommu_sva_handle_mm(struct iommu_fault *fault, struct mm_struct *mm)
 {
 	vm_fault_t ret;
 	struct vm_area_struct *vma;
-	struct mm_struct *mm = data;
 	unsigned int access_flags = 0;
 	unsigned int fault_flags = FAULT_FLAG_REMOTE;
 	struct iommu_fault_page_request *prm = &fault->prm;
Signature · 1 line
2.34.1
#11 lore raw

Re: [PATCH v5 09/12] iommu: Make iommu_queue_iopf() more generic

Quoted 17 lines · Lu Baolu

Make iommu_queue_iopf() more generic by making the iopf_group a minimal set of iopf's that an iopf handler of domain should handle and respond to. Add domain parameter to struct iopf_group so that the handler can retrieve and use it directly.

Change iommu_queue_iopf() to forward groups of iopf's to the domain's iopf handler. This is also a necessary step to decouple the sva iopf handling code from this interface.

---
  include/linux/iommu.h      |  4 ++--
  drivers/iommu/iommu-sva.h  |  6 ++---
  drivers/iommu/io-pgfault.c | 49 ++++++++++++++++++++++++++++----------
  drivers/iommu/iommu-sva.c  |  3 +--
  4 files changed, 42 insertions(+), 20 deletions(-)

......

Quoted 9 lines
drivers/iommu/io-pgfault.c +1 -0
@@ -112,6 +110,7 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
  {
  	int ret;
  	struct iopf_group *group;
+	struct iommu_domain *domain;
  	struct iopf_fault *iopf, *next;
  	struct iommu_fault_param *iopf_param;
  	struct dev_iommu *param = dev->iommu;

drivers/iommu/io-pgfault.c int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev) · line 150

Quoted 9 lines
drivers/iommu/io-pgfault.c +6 -0
@@ -143,6 +142,19 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
  		return 0;
  	}
 
+	if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)
+		domain = iommu_get_domain_for_dev_pasid(dev, fault->prm.pasid, 0);
+	else
+		domain = iommu_get_domain_for_dev(dev);
+
+	if (!domain || !domain->iopf_handler) {

Does it need to check if 'domain' is error ?  Like below:

         if (!domain || IS_ERR(domain) || !domain->iopf_handler) Thanks,
Jingqi

#12 lore raw

Re: [PATCH v5 09/12] iommu: Make iommu_queue_iopf() more generic

Quoted 41 lines · Jingqi
Quoted 17 lines · Lu Baolu

Make iommu_queue_iopf() more generic by making the iopf_group a minimal set of iopf's that an iopf handler of domain should handle and respond to. Add domain parameter to struct iopf_group so that the handler can retrieve and use it directly.

Change iommu_queue_iopf() to forward groups of iopf's to the domain's iopf handler. This is also a necessary step to decouple the sva iopf handling code from this interface.

---
  include/linux/iommu.h      |  4 ++--
  drivers/iommu/iommu-sva.h  |  6 ++---
  drivers/iommu/io-pgfault.c | 49 ++++++++++++++++++++++++++++----------
  drivers/iommu/iommu-sva.c  |  3 +--
  4 files changed, 42 insertions(+), 20 deletions(-)

......

Quoted 9 lines
drivers/iommu/io-pgfault.c +1 -0
@@ -112,6 +110,7 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
  {
  	int ret;
  	struct iopf_group *group;
+	struct iommu_domain *domain;
  	struct iopf_fault *iopf, *next;
  	struct iommu_fault_param *iopf_param;
  	struct dev_iommu *param = dev->iommu;

drivers/iommu/io-pgfault.c int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev) · line 149

Quoted 8 lines
drivers/iommu/io-pgfault.c +6 -0
@@ -143,6 +142,19 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
  		return 0;
  	}
+	if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)
+		domain = iommu_get_domain_for_dev_pasid(dev, fault->prm.pasid, 0);
+	else
+		domain = iommu_get_domain_for_dev(dev);
+
+	if (!domain || !domain->iopf_handler) {

Does it need to check if 'domain' is error ?  Like below:

Quoted 1 line

         if (!domain || IS_ERR(domain) || !domain->iopf_handler)

Urk, yes, but not like that

The IF needs to be moved into the else block as each individual function has its own return convention.

Jason

author maintainer
#13 lore raw

Re: [PATCH v5 09/12] iommu: Make iommu_queue_iopf() more generic

Quoted 42 lines · Jason Gunthorpe
Quoted 41 lines · Jingqi
Quoted 17 lines · Lu Baolu

Make iommu_queue_iopf() more generic by making the iopf_group a minimal set of iopf's that an iopf handler of domain should handle and respond to. Add domain parameter to struct iopf_group so that the handler can retrieve and use it directly.

Change iommu_queue_iopf() to forward groups of iopf's to the domain's iopf handler. This is also a necessary step to decouple the sva iopf handling code from this interface.

---
   include/linux/iommu.h      |  4 ++--
   drivers/iommu/iommu-sva.h  |  6 ++---
   drivers/iommu/io-pgfault.c | 49 ++++++++++++++++++++++++++++----------
   drivers/iommu/iommu-sva.c  |  3 +--
   4 files changed, 42 insertions(+), 20 deletions(-)

......

Quoted 9 lines
drivers/iommu/io-pgfault.c +1 -0
@@ -112,6 +110,7 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
   {
   	int ret;
   	struct iopf_group *group;
+	struct iommu_domain *domain;
   	struct iopf_fault *iopf, *next;
   	struct iommu_fault_param *iopf_param;
   	struct dev_iommu *param = dev->iommu;

drivers/iommu/io-pgfault.c int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev) · line 149

Quoted 8 lines
drivers/iommu/io-pgfault.c +6 -0
@@ -143,6 +142,19 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
   		return 0;
   	}
+	if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)
+		domain = iommu_get_domain_for_dev_pasid(dev, fault->prm.pasid, 0);
+	else
+		domain = iommu_get_domain_for_dev(dev);
+
+	if (!domain || !domain->iopf_handler) {

Does it need to check if 'domain' is error ?  Like below:

Quoted 6 lines
Quoted 1 line

         if (!domain || IS_ERR(domain) || !domain->iopf_handler)

Urk, yes, but not like that

The IF needs to be moved into the else block as each individual function has its own return convention.

iommu_get_domain_for_dev_pasid() returns an ERR_PTR only if the matching domain type is specified (non-zero).

Adding IS_ERR(domain) in the else block will make the code more readable. Alternatively we can put a comment around above code to explain that ERR_PTR is not a case here.

Best regards,
baolu

#14 lore raw

Re: [PATCH v5 09/12] iommu: Make iommu_queue_iopf() more generic

Quoted 27 lines · Baolu Lu
Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
drivers/iommu/io-pgfault.c +1 -0
@@ -112,6 +110,7 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
   {
   	int ret;
   	struct iopf_group *group;
+	struct iommu_domain *domain;
   	struct iopf_fault *iopf, *next;
   	struct iommu_fault_param *iopf_param;
   	struct dev_iommu *param = dev->iommu;

drivers/iommu/io-pgfault.c int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev) · line 149

Quoted 8 lines
drivers/iommu/io-pgfault.c +6 -0
@@ -143,6 +142,19 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
   		return 0;
   	}
+	if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)
+		domain = iommu_get_domain_for_dev_pasid(dev, fault->prm.pasid, 0);
+	else
+		domain = iommu_get_domain_for_dev(dev);
+
+	if (!domain || !domain->iopf_handler) {

Does it need to check if 'domain' is error ?  Like below:

Quoted 6 lines
Quoted 1 line

         if (!domain || IS_ERR(domain) || !domain->iopf_handler)

Urk, yes, but not like that

The IF needs to be moved into the else block as each individual function has its own return convention.

Quoted 6 lines

iommu_get_domain_for_dev_pasid() returns an ERR_PTR only if the matching domain type is specified (non-zero).

Adding IS_ERR(domain) in the else block will make the code more readable. Alternatively we can put a comment around above code to explain that ERR_PTR is not a case here.

You should check it because you'll probably get a static tool complaint otherwise

Jason

author maintainer
#15 lore raw

Re: [PATCH v5 09/12] iommu: Make iommu_queue_iopf() more generic

Quoted 26 lines · Jason Gunthorpe
Quoted 24 lines · Baolu Lu
Quoted 24 lines
Quoted 19 lines
Quoted 9 lines
drivers/iommu/io-pgfault.c +1 -0
@@ -112,6 +110,7 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
    {
    	int ret;
    	struct iopf_group *group;
+	struct iommu_domain *domain;
    	struct iopf_fault *iopf, *next;
    	struct iommu_fault_param *iopf_param;
    	struct dev_iommu *param = dev->iommu;

drivers/iommu/io-pgfault.c int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev) · line 149

Quoted 8 lines
drivers/iommu/io-pgfault.c +6 -0
@@ -143,6 +142,19 @@ int iommu_queue_iopf(struct iommu_fault *fault, struct device *dev)
    		return 0;
    	}
+	if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)
+		domain = iommu_get_domain_for_dev_pasid(dev, fault->prm.pasid, 0);
+	else
+		domain = iommu_get_domain_for_dev(dev);
+
+	if (!domain || !domain->iopf_handler) {

Does it need to check if 'domain' is error ?  Like below:

Quoted 1 line
           if (!domain || IS_ERR(domain) || !domain->iopf_handler)

Urk, yes, but not like that

The IF needs to be moved into the else block as each individual function has its own return convention.

Quoted 8 lines
Quoted 6 lines

iommu_get_domain_for_dev_pasid() returns an ERR_PTR only if the matching domain type is specified (non-zero).

Adding IS_ERR(domain) in the else block will make the code more readable. Alternatively we can put a comment around above code to explain that ERR_PTR is not a case here.

You should check it because you'll probably get a static tool complaint otherwise

Okay, got you.

Best regards,
baolu

Reviewed-by on v5 9/12
#16 lore raw

RE: [PATCH v5 09/12] iommu: Make iommu_queue_iopf() more generic

In reply to Lu Baolu
Quoted 9 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

Make iommu_queue_iopf() more generic by making the iopf_group a minimal
set of iopf's that an iopf handler of domain should handle and respond to. Add domain parameter to struct iopf_group so that the handler can retrieve and use it directly.

Quoted 5 lines

Change iommu_queue_iopf() to forward groups of iopf's to the domain's iopf handler. This is also a necessary step to decouple the sva iopf handling code from this interface.

With the fix of domain error check:

author maintainer
#17 lore raw

PATCH v5 10/12 iommu: Separate SVA and IOPF

In reply to Lu Baolu

Add CONFIG_IOMMU_IOPF for page fault handling framework and select it from its real consumer. Move iopf function declaration from iommu-sva.h to iommu.h and remove iommu-sva.h as it's empty now.

Consolidate all SVA related code into iommu-sva.c:
- Move iommu_sva_domain_alloc() from iommu.c to iommu-sva.c.
- Move sva iopf handling code from io-pgfault.c to iommu-sva.c.

Consolidate iommu_report_device_fault() and iommu_page_response() into io-pgfault.c.

Export iopf_free_group() for iopf handlers implemented in modules. Some functions are renamed with more meaningful names. No other intentional functionality changes.

---
 include/linux/iommu.h                         |  90 +++++++---
 drivers/iommu/iommu-sva.h                     |  69 --------
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |   1 -
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   1 -
 drivers/iommu/intel/iommu.c                   |   1 -
 drivers/iommu/intel/svm.c                     |   1 -
 drivers/iommu/io-pgfault.c                    | 163 ++++++++++++------
 drivers/iommu/iommu-sva.c                     |  79 ++++++++-
 drivers/iommu/iommu.c                         | 133 --------------
 drivers/iommu/Kconfig                         |   4 +
 drivers/iommu/Makefile                        |   3 +-
 drivers/iommu/intel/Kconfig                   |   1 +
 12 files changed, 257 insertions(+), 289 deletions(-)
 delete mode 100644 drivers/iommu/iommu-sva.h
include/linux/iommu.h +66 -24
index da47813c2e4c..1697ac168f05 100644
@@ -609,10 +609,6 @@ extern struct iommu_group *iommu_group_get(struct device *dev);
 extern struct iommu_group *iommu_group_ref_get(struct iommu_group *group);
 extern void iommu_group_put(struct iommu_group *group);
 
-extern int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt);
-extern int iommu_page_response(struct device *dev,
-			       struct iommu_page_response *msg);
-
 extern int iommu_group_id(struct iommu_group *group);
 extern struct iommu_domain *iommu_group_default_domain(struct iommu_group *);
 
@@ -802,8 +798,6 @@ bool iommu_group_dma_owner_claimed(struct iommu_group *group);
 int iommu_device_claim_dma_owner(struct device *dev, void *owner);
 void iommu_device_release_dma_owner(struct device *dev);
 
-struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
-					    struct mm_struct *mm);
 int iommu_attach_device_pasid(struct iommu_domain *domain,
 			      struct device *dev, ioasid_t pasid);
 void iommu_detach_device_pasid(struct iommu_domain *domain,
@@ -990,18 +984,6 @@ static inline void iommu_group_put(struct iommu_group *group)
 {
 }
 
-static inline
-int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt)
-{
-	return -ENODEV;
-}
-
-static inline int iommu_page_response(struct device *dev,
-				      struct iommu_page_response *msg)
-{
-	return -ENODEV;
-}
-
 static inline int iommu_group_id(struct iommu_group *group)
 {
 	return -ENODEV;
@@ -1138,12 +1120,6 @@ static inline int iommu_device_claim_dma_owner(struct device *dev, void *owner)
 	return -ENODEV;
 }
 
-static inline struct iommu_domain *
-iommu_sva_domain_alloc(struct device *dev, struct mm_struct *mm)
-{
-	return NULL;
-}
-
 static inline int iommu_attach_device_pasid(struct iommu_domain *domain,
 					    struct device *dev, ioasid_t pasid)
 {
@@ -1265,6 +1241,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev,
 					struct mm_struct *mm);
 void iommu_sva_unbind_device(struct iommu_sva *handle);
 u32 iommu_sva_get_pasid(struct iommu_sva *handle);
+struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
+					    struct mm_struct *mm);
 #else
 static inline struct iommu_sva *
 iommu_sva_bind_device(struct device *dev, struct mm_struct *mm)
@@ -1283,6 +1261,70 @@ static inline u32 iommu_sva_get_pasid(struct iommu_sva *handle)
 static inline void mm_pasid_init(struct mm_struct *mm) {}
 static inline bool mm_valid_pasid(struct mm_struct *mm) { return false; }
 static inline void mm_pasid_drop(struct mm_struct *mm) {}
+
+static inline struct iommu_domain *
+iommu_sva_domain_alloc(struct device *dev, struct mm_struct *mm)
+{
+	return NULL;
+}
 #endif /* CONFIG_IOMMU_SVA */
 
+#ifdef CONFIG_IOMMU_IOPF
+int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev);
+int iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev);
+int iopf_queue_flush_dev(struct device *dev);
+struct iopf_queue *iopf_queue_alloc(const char *name);
+void iopf_queue_free(struct iopf_queue *queue);
+int iopf_queue_discard_partial(struct iopf_queue *queue);
+void iopf_free_group(struct iopf_group *group);
+int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt);
+int iommu_page_response(struct device *dev, struct iommu_page_response *msg);
+#else
+static inline int
+iopf_queue_add_device(struct iopf_queue *queue, struct device *dev)
+{
+	return -ENODEV;
+}
+
+static inline int
+iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev)
+{
+	return -ENODEV;
+}
+
+static inline int iopf_queue_flush_dev(struct device *dev)
+{
+	return -ENODEV;
+}
+
+static inline struct iopf_queue *iopf_queue_alloc(const char *name)
+{
+	return NULL;
+}
+
+static inline void iopf_queue_free(struct iopf_queue *queue)
+{
+}
+
+static inline int iopf_queue_discard_partial(struct iopf_queue *queue)
+{
+	return -ENODEV;
+}
+
+static inline void iopf_free_group(struct iopf_group *group)
+{
+}
+
+static inline int
+iommu_report_device_fault(struct device *dev, struct iopf_fault *evt)
+{
+	return -ENODEV;
+}
+
+static inline int
+iommu_page_response(struct device *dev, struct iommu_page_response *msg)
+{
+	return -ENODEV;
+}
+#endif /* CONFIG_IOMMU_IOPF */
 #endif /* __LINUX_IOMMU_H */
drivers/iommu/iommu-sva.h deleted +0 -69

69 lines not on this page · raw message

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0 -1
index 4238c97ba07a..9b720334ede0 100644
@@ -10,7 +10,6 @@
 #include <linux/slab.h>
 
 #include "arm-smmu-v3.h"
-#include "../../iommu-sva.h"
 #include "../../io-pgtable-arm.h"
 
 struct arm_smmu_mmu_notifier {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -1
index e537b6c046c5..d0d349bfdba3 100644
@@ -29,7 +29,6 @@
 
 #include "arm-smmu-v3.h"
 #include "../../dma-iommu.h"
-#include "../../iommu-sva.h"
 
 static bool disable_bypass = true;
 module_param(disable_bypass, bool, 0444);
drivers/iommu/intel/iommu.c +0 -1
index 7b0ed6cdd6b9..e9b3f309273d 100644
@@ -27,7 +27,6 @@
 #include "iommu.h"
 #include "../dma-iommu.h"
 #include "../irq_remapping.h"
-#include "../iommu-sva.h"
 #include "pasid.h"
 #include "cap_audit.h"
 #include "perfmon.h"
drivers/iommu/intel/svm.c +0 -1
index 9de349ea215c..780c5bd73ec2 100644
@@ -22,7 +22,6 @@
 #include "iommu.h"
 #include "pasid.h"
 #include "perf.h"
-#include "../iommu-sva.h"
 #include "trace.h"
 
 static irqreturn_t prq_event_thread(int irq, void *d);
drivers/iommu/io-pgfault.c +111 -52

195 lines not on this page · raw message

drivers/iommu/iommu-sva.c +73 -6

100 lines not on this page · raw message

drivers/iommu/iommu.c +0 -133

151 lines not on this page · raw message

drivers/iommu/Kconfig +4 -0
index 2b12b583ef4b..e7b87ec525a7 100644
@@ -158,6 +158,9 @@ config IOMMU_DMA
 config IOMMU_SVA
 	bool
 
+config IOMMU_IOPF
+	bool
+
 config FSL_PAMU
 	bool "Freescale IOMMU support"
 	depends on PCI
@@ -404,6 +407,7 @@ config ARM_SMMU_V3_SVA
 	bool "Shared Virtual Addressing support for the ARM SMMUv3"
 	depends on ARM_SMMU_V3
 	select IOMMU_SVA
+	select IOMMU_IOPF
 	select MMU_NOTIFIER
 	help
 	  Support for sharing process address spaces with devices using the
drivers/iommu/Makefile +2 -1
index 769e43d780ce..6d114fe7ae89 100644
@@ -27,6 +27,7 @@ obj-$(CONFIG_FSL_PAMU) += fsl_pamu.o fsl_pamu_domain.o
 obj-$(CONFIG_S390_IOMMU) += s390-iommu.o
 obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o
 obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o
-obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o io-pgfault.o
+obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o
+obj-$(CONFIG_IOMMU_IOPF) += io-pgfault.o
 obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o
 obj-$(CONFIG_APPLE_DART) += apple-dart.o
drivers/iommu/intel/Kconfig +1 -0
index 2e56bd79f589..613f149510a7 100644
@@ -50,6 +50,7 @@ config INTEL_IOMMU_SVM
 	depends on X86_64
 	select MMU_NOTIFIER
 	select IOMMU_SVA
+	select IOMMU_IOPF
 	help
 	  Shared Virtual Memory (SVM) provides a facility for devices
 	  to access DMA resources through process address space by
Signature · 1 line
2.34.1
Reviewed-by on v5 10/12
#18 lore raw

RE: [PATCH v5 10/12] iommu: Separate SVA and IOPF

Quoted 14 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

Add CONFIG_IOMMU_IOPF for page fault handling framework and select it from its real consumer. Move iopf function declaration from iommu-sva.h to iommu.h and remove iommu-sva.h as it's empty now.

Consolidate all SVA related code into iommu-sva.c:
- Move iommu_sva_domain_alloc() from iommu.c to iommu-sva.c.
- Move sva iopf handling code from io-pgfault.c to iommu-sva.c.

Consolidate iommu_report_device_fault() and iommu_page_response() into io-pgfault.c.

Quoted 6 lines

Export iopf_free_group() for iopf handlers implemented in modules. Some functions are renamed with more meaningful names. No other intentional functionality changes.

author maintainer
#19 lore raw

PATCH v5 11/12 iommu: Consolidate per-device fault data management

In reply to Lu Baolu

The per-device fault data is a data structure that is used to store information about faults that occur on a device. This data is allocated when IOPF is enabled on the device and freed when IOPF is disabled. The data is used in the paths of iopf reporting, handling, responding, and draining.

The fault data is protected by two locks:

- dev->iommu->lock: This lock is used to protect the allocation and freeing of the fault data.
- dev->iommu->fault_parameter->lock: This lock is used to protect the fault data itself.

To make the code simpler and easier to maintain, consolidate the lock mechanism into two helper functions.

The dev->iommu->fault_parameter->lock lock is also used in iopf_queue_discard_partial() to improve code readability. This does not fix any existing issues, as iopf_queue_discard_partial() is only used in the VT-d driver's prq_event_thread(), which is a single-threaded path that reports the IOPFs.

---
 include/linux/iommu.h      |   3 +
 drivers/iommu/io-pgfault.c | 122 +++++++++++++++++++++++--------------
 2 files changed, 79 insertions(+), 46 deletions(-)
include/linux/iommu.h +3 -0
index 1697ac168f05..77ad33ffe3ac 100644
@@ -480,6 +480,8 @@ struct iommu_device {
 /**
  * struct iommu_fault_param - per-device IOMMU fault data
  * @lock: protect pending faults list
+ * @users: user counter to manage the lifetime of the data, this field
+ *         is protected by dev->iommu->lock.
  * @dev: the device that owns this param
  * @queue: IOPF queue
  * @queue_list: index into queue->devices
@@ -489,6 +491,7 @@ struct iommu_device {
  */
 struct iommu_fault_param {
 	struct mutex lock;
+	int users;
 
 	struct device *dev;
 	struct iopf_queue *queue;
drivers/iommu/io-pgfault.c +76 -46

224 lines not on this page · raw message

Signature · 1 line
2.34.1
Reviewed-by on v5 11/12
#20 lore raw

RE: [PATCH v5 11/12] iommu: Consolidate per-device fault data management

Quoted 13 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

drivers/iommu/io-pgfault.c +3 -0
+
+/* Caller must hold a reference of the fault parameter. */
+static void iopf_put_dev_fault_param(struct iommu_fault_param

*fault_param)

drivers/iommu/io-pgfault.c +6 -0
+{
+	struct device *dev = fault_param->dev;
+	struct dev_iommu *param = dev->iommu;
+
+	mutex_lock(&param->lock);
+	if (WARN_ON(fault_param->users <= 0 ||

drivers/iommu/io-pgfault.c void iopf_free_group(struct iopf_group *group) · line 68

Quoted 10 lines
drivers/iommu/io-pgfault.c +10 -0
+		    fault_param != param->fault_param)) {
+		mutex_unlock(&param->lock);
+		return;
+	}
+
+	if (--fault_param->users == 0) {
+		param->fault_param = NULL;
+		kfree(fault_param);
+		put_device(dev);
+	}

this could be a helper to be shared with iopf_queue_remove_device().

author maintainer
#21 lore raw

PATCH v5 12/12 iommu: Improve iopf_queue_flush_dev()

In reply to Lu Baolu

The iopf_queue_flush_dev() is called by the iommu driver before releasing a PASID. It ensures that all pending faults for this PASID have been handled or cancelled, and won't hit the address space that reuses this PASID. The driver must make sure that no new fault is added to the queue.

The SMMUv3 driver doesn't use it because it only implements the Arm-specific stall fault model where DMA transactions are held in the SMMU while waiting for the OS to handle iopf's. Since a device driver must complete all DMA transactions before detaching domain, there are no pending iopf's with the stall model. PRI support requires adding a call to iopf_queue_flush_dev() after flushing the hardware page fault queue.

The current implementation of iopf_queue_flush_dev() is a simplified version. It is only suitable for SVA case in which the processing of iopf is implemented in the inner loop of the iommu subsystem.

Improve this interface to make it also work for handling iopf out of the iommu core. Remove a warning message in iommu_page_response() since the iopf queue might get flushed before possible pending responses.

---
 include/linux/iommu.h      |  4 ++--
 drivers/iommu/intel/svm.c  |  2 +-
 drivers/iommu/io-pgfault.c | 46 +++++++++++++++++++++++++++++++++-----
 3 files changed, 44 insertions(+), 8 deletions(-)
include/linux/iommu.h +2 -2
index 77ad33ffe3ac..465e23e945d0 100644
@@ -1275,7 +1275,7 @@ iommu_sva_domain_alloc(struct device *dev, struct mm_struct *mm)
 #ifdef CONFIG_IOMMU_IOPF
 int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev);
 int iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev);
-int iopf_queue_flush_dev(struct device *dev);
+int iopf_queue_flush_dev(struct device *dev, ioasid_t pasid);
 struct iopf_queue *iopf_queue_alloc(const char *name);
 void iopf_queue_free(struct iopf_queue *queue);
 int iopf_queue_discard_partial(struct iopf_queue *queue);
@@ -1295,7 +1295,7 @@ iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev)
 	return -ENODEV;
 }
 
-static inline int iopf_queue_flush_dev(struct device *dev)
+static inline int iopf_queue_flush_dev(struct device *dev, ioasid_t pasid)
 {
 	return -ENODEV;
 }
drivers/iommu/intel/svm.c +1 -1
index 780c5bd73ec2..4c3f4533e337 100644
@@ -495,7 +495,7 @@ void intel_drain_pasid_prq(struct device *dev, u32 pasid)
 		goto prq_retry;
 	}
 
-	iopf_queue_flush_dev(dev);
+	iopf_queue_flush_dev(dev, pasid);
 
 	/*
 	 * Perform steps described in VT-d spec CH7.10 to drain page
drivers/iommu/io-pgfault.c +41 -5

74 lines not on this page · raw message

Signature · 1 line
2.34.1
#22 lore raw

RE: [PATCH v5 12/12] iommu: Improve iopf_queue_flush_dev()

Quoted 5 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

(quoted diff) +0 -0
@@ -300,6 +299,7 @@ EXPORT_SYMBOL_GPL(iommu_page_response);
 /**
  * iopf_queue_flush_dev - Ensure that all queued faults have been
Quoted 7 lines

processed
* @dev: the endpoint whose faults need to be flushed.
+ * @pasid: the PASID of the endpoint.
*
* The IOMMU driver calls this before releasing a PASID, to ensure that all
* pending faults for this PASID have been handled, and won't hit the
address

the comment should be updated too.

drivers/iommu/io-pgfault.c EXPORT_SYMBOL_GPL(iommu_page_response); · line 312

Quoted 6 lines
drivers/iommu/io-pgfault.c +1 -1
@@ -309,17 +309,53 @@ EXPORT_SYMBOL_GPL(iommu_page_response);
  *
  * Return: 0 on success and <0 on error.
  */
-int iopf_queue_flush_dev(struct device *dev)
+int iopf_queue_flush_dev(struct device *dev, ioasid_t pasid)

iopf_queue_flush_dev_pasid()?

Quoted 42 lines
drivers/iommu/io-pgfault.c +8 -0
 {
 	struct iommu_fault_param *iopf_param =
 iopf_get_dev_fault_param(dev);
+	const struct iommu_ops *ops = dev_iommu_ops(dev);
+	struct iommu_page_response resp;
+	struct iopf_fault *iopf, *next;
+	int ret = 0;
 
 	if (!iopf_param)
 		return -ENODEV;
 
 	flush_workqueue(iopf_param->queue->wq);
+
+	mutex_lock(&iopf_param->lock);
+	list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) {
+		if (!(iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) ||

drivers/iommu/io-pgfault.c +9 -0
+		    iopf->fault.prm.pasid != pasid)
+			break;
+
+		list_del(&iopf->list);
+		kfree(iopf);
+	}
+
+	list_for_each_entry_safe(iopf, next, &iopf_param->faults, list) {
+		if (!(iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) ||

drivers/iommu/io-pgfault.c +9 -0
+		    iopf->fault.prm.pasid != pasid)
+			continue;
+
+		memset(&resp, 0, sizeof(struct iommu_page_response));
+		resp.pasid = iopf->fault.prm.pasid;
+		resp.grpid = iopf->fault.prm.grpid;
+		resp.code = IOMMU_PAGE_RESP_INVALID;
+
+		if (iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID)

drivers/iommu/io-pgfault.c +5 -0
+			resp.flags = IOMMU_PAGE_RESP_PASID_VALID;
+
+		ret = ops->page_response(dev, iopf, &resp);
+		if (ret)
+			break;

drivers/iommu/io-pgfault.c EXPORT_SYMBOL_GPL(iommu_page_response); · line 359

Quoted 10 lines
drivers/iommu/io-pgfault.c +6 -1
+
+		list_del(&iopf->list);
+		kfree(iopf);
+	}
+	mutex_unlock(&iopf_param->lock);
 	iopf_put_dev_fault_param(iopf_param);
 
-	return 0;
+	return ret;
 }

Is it more accurate to call this function as iopf_queue_drop_dev_pasid()? The added logic essentially implies that the caller doesn't care about responses and all the in-fly states are either flushed (request) or abandoned (response).

A normal flush() helper usually means just the flush action. If there is a need to wait for responses after flush then we could add a flush_dev_pasid_wait_timeout() later when there is a demand...

author maintainer
#23 lore raw

Re: [PATCH v5 12/12] iommu: Improve iopf_queue_flush_dev()

Quoted 6 lines · Kevin
Quoted 6 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

(quoted diff) +0 -0
@@ -300,6 +299,7 @@ EXPORT_SYMBOL_GPL(iommu_page_response);
  /**
   * iopf_queue_flush_dev - Ensure that all queued faults have been

processed

Quoted 8 lines
Quoted 6 lines

* @dev: the endpoint whose faults need to be flushed.
+ * @pasid: the PASID of the endpoint.
*
* The IOMMU driver calls this before releasing a PASID, to ensure that all
* pending faults for this PASID have been handled, and won't hit the
address

the comment should be updated too.

Yes.

... pending faults for this PASID have been handled or dropped ...

Quoted 68 lines

drivers/iommu/io-pgfault.c EXPORT_SYMBOL_GPL(iommu_page_response); · line 312

Quoted 7 lines
drivers/iommu/io-pgfault.c +1 -1
@@ -309,17 +309,53 @@ EXPORT_SYMBOL_GPL(iommu_page_response);
   *
   * Return: 0 on success and <0 on error.
   */
-int iopf_queue_flush_dev(struct device *dev)
+int iopf_queue_flush_dev(struct device *dev, ioasid_t pasid)

iopf_queue_flush_dev_pasid()?

Quoted 42 lines
drivers/iommu/io-pgfault.c +8 -0
  {
  	struct iommu_fault_param *iopf_param =
 iopf_get_dev_fault_param(dev);
+	const struct iommu_ops *ops = dev_iommu_ops(dev);
+	struct iommu_page_response resp;
+	struct iopf_fault *iopf, *next;
+	int ret = 0;
 
  	if (!iopf_param)
  		return -ENODEV;
 
  	flush_workqueue(iopf_param->queue->wq);
+
+	mutex_lock(&iopf_param->lock);
+	list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) {
+		if (!(iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) ||

drivers/iommu/io-pgfault.c +9 -0
+		    iopf->fault.prm.pasid != pasid)
+			break;
+
+		list_del(&iopf->list);
+		kfree(iopf);
+	}
+
+	list_for_each_entry_safe(iopf, next, &iopf_param->faults, list) {
+		if (!(iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) ||

drivers/iommu/io-pgfault.c +9 -0
+		    iopf->fault.prm.pasid != pasid)
+			continue;
+
+		memset(&resp, 0, sizeof(struct iommu_page_response));
+		resp.pasid = iopf->fault.prm.pasid;
+		resp.grpid = iopf->fault.prm.grpid;
+		resp.code = IOMMU_PAGE_RESP_INVALID;
+
+		if (iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID)

drivers/iommu/io-pgfault.c +5 -0
+			resp.flags = IOMMU_PAGE_RESP_PASID_VALID;
+
+		ret = ops->page_response(dev, iopf, &resp);
+		if (ret)
+			break;

drivers/iommu/io-pgfault.c EXPORT_SYMBOL_GPL(iommu_page_response); · line 359

Quoted 10 lines
drivers/iommu/io-pgfault.c +6 -1
+
+		list_del(&iopf->list);
+		kfree(iopf);
+	}
+	mutex_unlock(&iopf_param->lock);
  	iopf_put_dev_fault_param(iopf_param);
 
-	return 0;
+	return ret;
  }

Is it more accurate to call this function as iopf_queue_drop_dev_pasid()? The added logic essentially implies that the caller doesn't care about responses and all the in-fly states are either flushed (request) or abandoned (response).

Quoted 3 lines

A normal flush() helper usually means just the flush action. If there is a need to wait for responses after flush then we could add a flush_dev_pasid_wait_timeout() later when there is a demand...

Fair enough.

As my understanding, "flush" means "handling the pending i/o page faults immediately and wait until everything is done". Here what the caller wants is "I have completed using this pasid, discard all the pending requests by responding an INVALID result so that this PASID could be reused".

If this holds, how about iopf_queue_discard_dev_pasid()? It matches the existing iopf_queue_discard_partial().

Best regards,
baolu

#24 lore raw

RE: [PATCH v5 12/12] iommu: Improve iopf_queue_flush_dev()

Quoted 97 lines

From: Baolu Lu <[email protected]>
Sent: Tuesday, September 26, 2023 9:49 AM

Quoted 6 lines · Kevin
Quoted 6 lines

From: Lu Baolu <[email protected]>
Sent: Thursday, September 14, 2023 4:57 PM

(quoted diff) +0 -0
@@ -300,6 +299,7 @@ EXPORT_SYMBOL_GPL(iommu_page_response);
  /**
   * iopf_queue_flush_dev - Ensure that all queued faults have been

processed

Quoted 8 lines
Quoted 6 lines

* @dev: the endpoint whose faults need to be flushed.
+ * @pasid: the PASID of the endpoint.
*
* The IOMMU driver calls this before releasing a PASID, to ensure that all
* pending faults for this PASID have been handled, and won't hit the
address

the comment should be updated too.

Yes.

... pending faults for this PASID have been handled or dropped ...

Quoted 68 lines

drivers/iommu/io-pgfault.c EXPORT_SYMBOL_GPL(iommu_page_response); · line 312

Quoted 7 lines
drivers/iommu/io-pgfault.c +1 -1
@@ -309,17 +309,53 @@ EXPORT_SYMBOL_GPL(iommu_page_response);
   *
   * Return: 0 on success and <0 on error.
   */
-int iopf_queue_flush_dev(struct device *dev)
+int iopf_queue_flush_dev(struct device *dev, ioasid_t pasid)

iopf_queue_flush_dev_pasid()?

Quoted 42 lines
drivers/iommu/io-pgfault.c +8 -0
  {
  	struct iommu_fault_param *iopf_param =
 iopf_get_dev_fault_param(dev);
+	const struct iommu_ops *ops = dev_iommu_ops(dev);
+	struct iommu_page_response resp;
+	struct iopf_fault *iopf, *next;
+	int ret = 0;
 
  	if (!iopf_param)
  		return -ENODEV;
 
  	flush_workqueue(iopf_param->queue->wq);
+
+	mutex_lock(&iopf_param->lock);
+	list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) {
+		if (!(iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) ||

drivers/iommu/io-pgfault.c +9 -0
+		    iopf->fault.prm.pasid != pasid)
+			break;
+
+		list_del(&iopf->list);
+		kfree(iopf);
+	}
+
+	list_for_each_entry_safe(iopf, next, &iopf_param->faults, list) {
+		if (!(iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) ||

drivers/iommu/io-pgfault.c +9 -0
+		    iopf->fault.prm.pasid != pasid)
+			continue;
+
+		memset(&resp, 0, sizeof(struct iommu_page_response));
+		resp.pasid = iopf->fault.prm.pasid;
+		resp.grpid = iopf->fault.prm.grpid;
+		resp.code = IOMMU_PAGE_RESP_INVALID;
+
+		if (iopf->fault.prm.flags &

IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID)

drivers/iommu/io-pgfault.c +5 -0
+			resp.flags = IOMMU_PAGE_RESP_PASID_VALID;
+
+		ret = ops->page_response(dev, iopf, &resp);
+		if (ret)
+			break;

drivers/iommu/io-pgfault.c EXPORT_SYMBOL_GPL(iommu_page_response); · line 359

Quoted 10 lines
drivers/iommu/io-pgfault.c +6 -1
+
+		list_del(&iopf->list);
+		kfree(iopf);
+	}
+	mutex_unlock(&iopf_param->lock);
  	iopf_put_dev_fault_param(iopf_param);
 
-	return 0;
+	return ret;
  }

Is it more accurate to call this function as iopf_queue_drop_dev_pasid()? The added logic essentially implies that the caller doesn't care about responses and all the in-fly states are either flushed (request) or abandoned (response).

Quoted 3 lines

A normal flush() helper usually means just the flush action. If there is a need to wait for responses after flush then we could add a flush_dev_pasid_wait_timeout() later when there is a demand...

Fair enough.

Quoted 9 lines

As my understanding, "flush" means "handling the pending i/o page faults immediately and wait until everything is done". Here what the caller wants is "I have completed using this pasid, discard all the pending requests by responding an INVALID result so that this PASID could be reused".

If this holds, how about iopf_queue_discard_dev_pasid()? It matches the existing iopf_queue_discard_partial().

yes. 'discard' sounds better.