iommufd: Iterate the cache invalidation array in the core

The arm_vsmmu_cache_invalidate() op hands a guest's invalidation commands to the trusted main command queue after enforcing only the VMID or the SID, and passes the rest of the command through to the queue unchanged.

That lets a guest set bits the host never meant to forward: a reserved or undefined bit makes a command malformed; per the Arm SMMUv3 specification, in its section 4.1.3 "Command errors", a CERROR_ILL is raised, among other cases, when:

A valid command opcode is used and a Reserved or undefined field is optionally detected as non-zero, which results in the command being treated as malformed.

Restrict each opcode to the fields that the driver supports and reject the command with -EIO if it sets any other bit, before the command reaches the queue. This stops the host from forwarding any bit whose meaning it does not control. Document this contract in the uAPI header, so user space must take the responsibility to forward valid commands only.

Some fields and whole opcodes are legal only on an SMMU that implements the matching feature, so accept them conditionally:

- NUM, SCALE, TG and TTL need FEAT_RANGE_INV.
- SCALE bit 25, for values above 31, and TTL == 0b01 with a 16KB TG need SMMU_IDR5.DS. The raw IDR5 that iommufd reports has let a VMM expose DS to its guest all along, so add the minimal ARM_SMMU_FEAT_DS detection and gate the two encodings on it, rather than regress such a VMM by rejecting them unconditionally. Widening the SCALE field does not change what the host emits in the range invalidation path, which now masks its scale value explicitly to keep the pre-existing 5-bit truncation.
- ASID is limited to asid_bits, since its upper 8 bits are RES0 on an SMMU that only supports 8-bit ASIDs.
- ATC_INV needs FEAT_ATS. Per the specification's section 4.5 "ATS and PRI", CMD_ATC_INV is ILLEGAL when:

SMMU_IDR0.ATS == 0 and this command is issued on a Non-secure or Secure Command queue.

- SSV, SSID and Global need a non-zero ssid_bits. Without it, setting them is not illegal but CONSTRAINED UNPREDICTABLE, which a guest should not be able to provoke. Global also takes effect only when SSV == 1, broadening the invalidation from the one SSID to all the PASIDs of the single device that the SID field addresses.

Some values inside the accepted fields are Reserved too:

- NUM == 0, SCALE == 0 and TTL == 0 together are a Reserved combination and cause a CERROR_ILL.
- NUM, SCALE and TTL turn RES0 when TG == 0.
- An ATC_INV Size above 52, the invalidate-all span, is permitted to raise a CERROR_ILL.

Reject these Reserved values the same way. In contrast, an out-of-range address or ID value is defined as CONSTRAINED UNPREDICTABLE that would be scoped to the guest itself, so it does not deserve a check.

Fixes d68beb276ba2 ("iommu/arm-smmu-v3: Support IOMMU_HWPT_INVALIDATE using a VIOMMU object")

Cc [email protected]

Assisted-by Claude:claude-fable-5

Signed-off-by Nicolin Chen <[email protected]>

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   4 +-
 include/uapi/linux/iommufd.h                  |   4 +-
 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 116 ++++++++++++++++--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   6 +-
 4 files changed, 116 insertions(+), 14 deletions(-)

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +2 -1

index c909c9a88538b..3c59e62978a13 100644
@@ -64,6 +64,7 @@ struct arm_vsmmu;
 
 #define ARM_SMMU_IDR5			0x14
 #define IDR5_STALL_MAX			GENMASK(31, 16)
+#define IDR5_DS				(1 << 7)
 #define IDR5_GRAN64K			(1 << 6)
 #define IDR5_GRAN16K			(1 << 5)
 #define IDR5_GRAN4K			(1 << 4)

@@ -415,7 +416,7 @@ struct arm_smmu_cmd {
 
 #define CMDQ_TLBI_0_NUM			GENMASK_ULL(16, 12)
 #define CMDQ_TLBI_RANGE_NUM_MAX		31
-#define CMDQ_TLBI_0_SCALE		GENMASK_ULL(24, 20)
+#define CMDQ_TLBI_0_SCALE		GENMASK_ULL(25, 20)
 #define CMDQ_TLBI_0_VMID		GENMASK_ULL(47, 32)
 #define CMDQ_TLBI_0_ASID		GENMASK_ULL(63, 48)
 #define CMDQ_TLBI_1_LEAF		(1UL << 0)

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -0

 #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)

include/uapi/linux/iommufd.h +3 -1

index 0425d452d41ed..691b810ca574f 100644
@@ -909,7 +909,9 @@ struct iommu_hwpt_vtd_s1_invalidate {
  *     CMDQ_OP_CFGI_CD
  *     CMDQ_OP_CFGI_CD_ALL
  *
- * -EIO will be returned if the command is not supported.
+ * User space must forward only valid commands: the kernel rejects, with
+ * -EIO, any command carrying an unsupported opcode, an unsupported field,
+ * or a field value that the underlying SMMU hardware does not implement.
  */
 struct iommu_viommu_arm_smmuv3_invalidate {
 	__aligned_le64 cmd[2];

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0 -0

index 1e9f7d2de3441..a5cfbc2499034 100644

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +74 -2

+		return -EIO;
+	/* NUM == 0, SCALE == 0 with TTL == 0 is a reserved combination */
+	if (!range && !ttl)
+		return -EIO;
+	return 0;
+}
+
 /*
  * Convert, in place, the raw invalidation command into an internal format that
  * can be passed to arm_smmu_cmdq_issue_cmdlist(). Internally commands are

@@ -315,33 +335,107 @@ struct arm_vsmmu_invalidation_cmd {
 static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
 				      struct arm_vsmmu_invalidation_cmd *cmd)
 {
+	struct arm_smmu_device *smmu = vsmmu->smmu;
+	u64 allowed[2] = { CMDQ_0_OP };
+	u64 *data = cmd->cmd.data;
+
 	/* Commands are le64 stored in u64 */
-	cmd->cmd.data[0] = le64_to_cpu(cmd->ucmd.cmd[0]);
-	cmd->cmd.data[1] = le64_to_cpu(cmd->ucmd.cmd[1]);
+	data[0] = le64_to_cpu(cmd->ucmd.cmd[0]);
+	data[1] = le64_to_cpu(cmd->ucmd.cmd[1]);
+
+	/* Collect the fields userspace is allowed to set for each opcode */
+	switch (data[0] & CMDQ_0_OP) {
+	case CMDQ_OP_TLBI_NH_VA:
+		/* An SMMU with 8-bit ASIDs treats the upper 8 bits as RES0 */
+		allowed[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID,
+					 GENMASK(smmu->asid_bits - 1, 0));
+		fallthrough;
+	case CMDQ_OP_TLBI_NH_VAA:
+		/* NUM/SCALE/TG/TTL are range fields gated on FEAT_RANGE_INV */
+		if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
+			if (arm_vsmmu_validate_range(smmu, cmd))
+				return -EIO;
+			allowed[0] |= CMDQ_TLBI_0_NUM | CMDQ_TLBI_0_SCALE;
+			allowed[1] |= CMDQ_TLBI_1_TG | CMDQ_TLBI_1_TTL;
+			/* SCALE bit 25 (values above 31) is RES0 without DS */
+			if (!(smmu->features & ARM_SMMU_FEAT_DS))
+				allowed[0] &= ~FIELD_PREP(CMDQ_TLBI_0_SCALE,
+							  BIT(5));
+		}
+		allowed[0] |= CMDQ_TLBI_0_VMID;
+		allowed[1] |= CMDQ_TLBI_1_LEAF | CMDQ_TLBI_1_VA_MASK;
+		break;
+	case CMDQ_OP_TLBI_NH_ASID:
+		/* An SMMU with 8-bit ASIDs treats the upper 8 bits as RES0 */
+		allowed[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID,
+					 GENMASK(smmu->asid_bits - 1, 0));
+		fallthrough;
+	case CMDQ_OP_TLBI_NH_ALL:
+		allowed[0] |= CMDQ_TLBI_0_VMID;
+		break;
+	case CMDQ_OP_ATC_INV:
+		/* ATC_INV is illegal unless the SMMU implements ATS */
+		if (!(smmu->features & ARM_SMMU_FEAT_ATS))
+			return -EIO;
+		/* A Size above 52 (invalidate-all) may raise a CERROR_ILL */
+		if (FIELD_GET(CMDQ_ATC_1_SIZE, data[1]) > ATC_INV_SIZE_ALL)
+			return -EIO;
+		/*
+		 * SSV/SSID/Global need substream support. SSID and Global are
+		 * IGNORED (not RES0) when SSV == 0, so they need no SSV check.
+		 */
+		if (smmu->ssid_bits)
+			allowed[0] |= CMDQ_0_SSV | CMDQ_ATC_0_SSID |
+				      CMDQ_ATC_0_GLOBAL;
+		allowed[0] |= CMDQ_ATC_0_SID;
+		allowed[1] |= CMDQ_ATC_1_SIZE | CMDQ_ATC_1_ADDR_MASK;
+		break;
+	case CMDQ_OP_CFGI_CD:
+		/* No SSV for CFGI_CD; SSID requires substream support */
+		if (smmu->ssid_bits)
+			allowed[0] |= CMDQ_CFGI_0_SSID;
+		allowed[1] |= CMDQ_CFGI_1_LEAF;
+		fallthrough;
+	case CMDQ_OP_CFGI_CD_ALL:
+		allowed[0] |= CMDQ_CFGI_0_SID;
+		break;
+	}
+
+	/*
+	 * Reject any other bit, e.g. a RES0 bit or a Secure bit, before the

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +9 -9

-	switch (cmd->cmd.data[0] & CMDQ_0_OP) {
+	switch (data[0] & CMDQ_0_OP) {
 	case CMDQ_OP_TLBI_NSNH_ALL:
 		/* Convert to NH_ALL */
-		cmd->cmd.data[0] = CMDQ_OP_TLBI_NH_ALL |
-			      FIELD_PREP(CMDQ_TLBI_0_VMID, vsmmu->vmid);
-		cmd->cmd.data[1] = 0;
+		data[0] = CMDQ_OP_TLBI_NH_ALL |
+			  FIELD_PREP(CMDQ_TLBI_0_VMID, vsmmu->vmid);
+		data[1] = 0;
 		break;
 	case CMDQ_OP_TLBI_NH_VA:
 	case CMDQ_OP_TLBI_NH_VAA:
 	case CMDQ_OP_TLBI_NH_ALL:
 	case CMDQ_OP_TLBI_NH_ASID:
-		cmd->cmd.data[0] &= ~CMDQ_TLBI_0_VMID;
-		cmd->cmd.data[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, vsmmu->vmid);
+		data[0] &= ~CMDQ_TLBI_0_VMID;
+		data[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, vsmmu->vmid);
 		break;
 	case CMDQ_OP_ATC_INV:
 	case CMDQ_OP_CFGI_CD:
 	case CMDQ_OP_CFGI_CD_ALL: {
-		u32 sid, vsid = FIELD_GET(CMDQ_CFGI_0_SID, cmd->cmd.data[0]);
+		u32 sid, vsid = FIELD_GET(CMDQ_CFGI_0_SID, data[0]);
 
 		if (arm_vsmmu_vsid_to_sid(vsmmu, vsid, &sid))
 			return -EIO;
-		cmd->cmd.data[0] &= ~CMDQ_CFGI_0_SID;
-		cmd->cmd.data[0] |= FIELD_PREP(CMDQ_CFGI_0_SID, sid);
+		data[0] &= ~CMDQ_CFGI_0_SID;
+		data[0] |= FIELD_PREP(CMDQ_CFGI_0_SID, sid);
 		break;
 	}
 	default:

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -1

index a10affb483a4f..9f121f9f404ea 100644
@@ -2446,9 +2446,10 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
 			/* Determine how many chunks of 2^scale size we have */
 			num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
 
+			/* Keep the pre-DS 5-bit truncation when scale > 31 */
 			cmd->data[0] = orig_data0 |
 				FIELD_PREP(CMDQ_TLBI_0_NUM, num - 1) |
-				FIELD_PREP(CMDQ_TLBI_0_SCALE, scale);
+				FIELD_PREP(CMDQ_TLBI_0_SCALE, scale & 0x1f);
 
 			/* range is num * 2^scale * pgsize */
 			inv_range = num << (scale + tg);

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0

 	/* Page sizes */
 	if (reg & IDR5_GRAN64K)
 		smmu->pgsize_bitmap |= SZ_64K | SZ_512M;

Signature · 1 line

2.43.0

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c struct arm_vsmmu_invalidation_cmd { · line 320

Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +8 -0

+	u8 ttl = FIELD_GET(CMDQ_TLBI_1_TTL, cmd->cmd.data[1]);
+	u8 tg = FIELD_GET(CMDQ_TLBI_1_TG, cmd->cmd.data[1]);
+
+	/* NUM, SCALE and TTL are RES0 when TG == 0 */
+	if (!tg)
+		return (range || ttl) ? -EIO : 0;
+	/* TTL == 0b01 with a 16KB TG requires SMMU_IDR5.DS */
+	if (tg == 2 && ttl == 1 && !(smmu->features & ARM_SMMU_FEAT_DS))

I agree we must check this but I'm a little confused, How is a user / VMM depending on the IDR5.DS bit today? The IDR5 is exposed to the user via hw_info AND in the uAPI documentation block we explicitly mention for struct iommu_hw_info_arm_smmuv3 the following valid fields for IDR5:

idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K

Quoted 14 lines · Pranjal Shrivastava

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c struct arm_vsmmu_invalidation_cmd { · line 320

Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +8 -0

+	u8 ttl = FIELD_GET(CMDQ_TLBI_1_TTL, cmd->cmd.data[1]);
+	u8 tg = FIELD_GET(CMDQ_TLBI_1_TG, cmd->cmd.data[1]);
+
+	/* NUM, SCALE and TTL are RES0 when TG == 0 */
+	if (!tg)
+		return (range || ttl) ? -EIO : 0;
+	/* TTL == 0b01 with a 16KB TG requires SMMU_IDR5.DS */
+	if (tg == 2 && ttl == 1 && !(smmu->features & ARM_SMMU_FEAT_DS))

I agree we must check this but I'm a little confused, How is a user / VMM depending on the IDR5.DS bit today? The IDR5 is exposed to the user via hw_info AND in the uAPI documentation block we explicitly mention for struct iommu_hw_info_arm_smmuv3 the following valid fields for IDR5:

Quoted 1 line

idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K

Probably soon we will add DS to that comment and it will be fully backwards compatible so a new VMM following the new guidance can just read DS.

Thus we can't have any kernel release that doesn't work with a VMM following DS...

Regarding this series, since we are formally adding support for the DS bit in this series for range invalidations (preferably as a separate patch as mentioned above). We should also update the uAPI doc for struct iommu_hw_info_arm_smmuv3 in include/uapi/linux/iommufd.h? Like:

The cache invalidation ops, cache_invalidate_user() for a nested HWPT and the cache_invalidate() for a vIOMMU, are each handed the full user request array and report how many of the array entries they handled by setting the array->entry_num. Every driver therefore implements its own loop over the array, and a driver wanting to process that array in fixed-size chunks (e.g. to issue commands out of a fixed-size on-stack buffer) has to carry the loop and its sub-array bookkeeping all on its own.

Move the iteration into the iommufd core instead. Invoke the op with a sub-array that starts at the first not-yet-handled entry, let it handle a prefix of that sub-array and report the count via array->entry_num, then advance the base pointer and re-invoke the op until the entire array has been consumed or until the op returns an error along the way.

A driver that handles the entire window in one single call, as all of the current drivers happen to do, finishes the loop in just one pass, so this does not change any of the existing behavior. It instead lets each of the drivers convert to bounded chunk processing on its own, done by each of the subsequent changes.

Suggested-by Jason Gunthorpe<[email protected]>

Reviewed-by Kevin Tian<[email protected]>

Assisted-by Claude:claude-opus-4-8

Signed-off-by Nicolin Chen<[email protected]>

---
  include/linux/iommu.h                |  6 ++++--
  include/linux/iommufd.h              |  2 ++
  drivers/iommu/iommufd/hw_pagetable.c | 25 +++++++++++++++----------
  3 files changed, 21 insertions(+), 12 deletions(-)

The cache invalidation ops, cache_invalidate_user() for a nested HWPT and the cache_invalidate() for a vIOMMU, are each handed the full user request array and report how many of the array entries they handled by setting the array->entry_num. Every driver therefore implements its own loop over the array, and a driver wanting to process that array in fixed-size chunks (e.g. to issue commands out of a fixed-size on-stack buffer) has to carry the loop and its sub-array bookkeeping all on its own.

Move the iteration into the iommufd core instead. Invoke the op with a sub-array that starts at the first not-yet-handled entry, let it handle a prefix of that sub-array and report the count via array->entry_num, then advance the base pointer and re-invoke the op until the entire array has been consumed or until the op returns an error along the way.

A driver that handles the entire window in one single call, as all of the current drivers happen to do, finishes the loop in just one pass, so this does not change any of the existing behavior. It instead lets each of the drivers convert to bounded chunk processing on its own, done by each of the subsequent changes.

Suggested-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

Assisted-by Claude:claude-opus-4-8

Signed-off-by Nicolin Chen <[email protected]>

The vIOMMU and the nested-domain selftest invalidation mocks each used to walk the whole request array on their own, with the vIOMMU mock even allocating a buffer sized to the entire array in order to do so first.

The iommufd core now iterates the request array itself and re-invokes the op with the not-yet-handled sub-array, so handle just a single request per call out of the front of that sub-array and report one handled entry via the array->entry_num. Drop both of the loops and the kzalloc_objs() in the viommu callback function, and keep returning a success for an empty array as a probe of the selftest data type.

Reviewed-by Kevin Tian <[email protected]>

Assisted-by Claude:claude-opus-4-8

Signed-off-by Nicolin Chen <[email protected]>

arm_vsmmu_cache_invalidate() allocated a buffer for the entire user request array, walked the array converting each of the commands, and issued those converted commands to the cmdq in CMDQ_BATCH_ENTRIES sized chunks, carrying the sub-array bookkeeping all on its own.

The iommufd core now iterates the invalidation array and re-invokes the op with the not-yet-handled sub-array, so the driver only has to proceed with a single chunk per call.

Instead of a per-array allocation, use a fixed on-stack batch to copy from the userspace array. If the copy fails due to nonzero padding (VMM violates the ABI), fail the entire batch.

Convert the whole batch before issuing any of it: a malformed command is a userspace bug, so the first illegal command fails the batch as a unit, issuing nothing and leaving array->entry_num at zero, the same way the copy above bails on nonzero padding. A batch that converts cleanly is issued in full, so the op returns either a handled count with no error or zero with an error.

A zero-length array now returns success once the data type gets validated, matching the documented probe behavior, rather than the -EINVAL that the full-array copy helper would previously return.

This also fixes two long-standing bugs:
1) On a conversion failure the old code reported commands that it had converted but not yet issued, so user space advanced its consumer index past invalidations that never reached the cmdq.
2) A zero-length array was rejected with -EINVAL, although the uAPI documents it as a valid request that only probes the data type.

Reviewed-by Kevin Tian <[email protected]>

Assisted-by Claude:claude-opus-4-8

Signed-off-by Nicolin Chen <[email protected]>

intel_nested_cache_invalidate_user() used to walk the whole request array on its own, copying and then flushing one single entry at a time.

The iommufd core now iterates the request array itself and re-invokes the op with the not-yet-handled sub-array, so handle just a single request per call out of the front of that sub-array and report one handled entry via the array->entry_num. An empty array keeps returning a success, used as a probe of IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 support.

Reviewed-by Kevin Tian<[email protected]>

Assisted-by Claude:claude-opus-4-8

Signed-off-by Nicolin Chen<[email protected]>

---
  drivers/iommu/intel/nested.c | 54 ++++++++++++++++++++----------------
  1 file changed, 30 insertions(+), 24 deletions(-)