Skip to contentSkip to conversation

iommu/arm-smmu-v3: Introduce an RCU-protected invalidation array

RFC PATCH Superseded Nicolin Chen 4 participants 35 messages Sep 8, 2025 – Oct 1, 2025

Subsystems (3)

v2 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#12 lore raw

RFC PATCH v2 3/8 iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free()

In reply to Nicolin Chen

There will be a bit more things to free than smmu_domain itself. So keep a simple inline function in the header to share aross files.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 5 +++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 4 ++--
 3 files changed, 8 insertions(+), 3 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +5 -0
index 5c0b38595d209..96a23ca633cb6 100644
@@ -954,6 +954,11 @@ extern struct mutex arm_smmu_asid_lock;
 
 struct arm_smmu_domain *arm_smmu_domain_alloc(void);
 
+static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domain)
+{
+	kfree(smmu_domain);
+}
+
 void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
 struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
 					u32 ssid);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -1
index 6097f1f540d87..fc601b494e0af 100644
@@ -365,6 +365,6 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
 err_asid:
 	xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
 err_free:
-	kfree(smmu_domain);
+	arm_smmu_domain_free(smmu_domain);
 	return ERR_PTR(ret);
 }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -2
index 0016ec699acfe..08af5f2d1235a 100644
@@ -2498,7 +2498,7 @@ static void arm_smmu_domain_free_paging(struct iommu_domain *domain)
 			ida_free(&smmu->vmid_map, cfg->vmid);
 	}
 
-	kfree(smmu_domain);
+	arm_smmu_domain_free(smmu_domain);
 }
 
 static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
@@ -3359,7 +3359,7 @@ arm_smmu_domain_alloc_paging_flags(struct device *dev, u32 flags,
 	return &smmu_domain->domain;
 
 err_free:
-	kfree(smmu_domain);
+	arm_smmu_domain_free(smmu_domain);
 	return ERR_PTR(ret);
 }
 
Signature · 1 line
2.43.0
#0 lore raw

v2 cover iommu/arm-smmu-v3: Introduce an RCU-protected invalidation array

This is a work based on Jason's design and algorithm. This implementation follows his initial draft and revising as well.

The new arm_smmu_invs array is an RCU-protected array, mutated when device attach to the domain, iterated when an invalidation is required for IOPTE changes in this domain. This keeps the current invalidation efficiency of a smb_mb() followed by a conditional rwlock replacing the atomic/spinlock combination.

A new data structure is defined for the array and its entry, representing invalidation operations, such as S1_ASID, S2_VMID, and ATS. The algorithm adds and deletes array entries efficiently and also keeps the array sorted so as to group similar invalidations into batches.

During an invalidation, a new invalidation function iterates domain->invs, and converts each entry to the corresponding invalidation command(s). This new function is fully compatible with all the existing use cases, allowing a simple rework/replacement.

Some races to keep in mind:

1) A domain can be shared across SMMU instances. When an SMMU instance is removed, the updated invs array has to be sync-ed via synchronize_rcu() to prevent an concurrent invalidation routine that is accessing the old array from issuing commands to the removed SMMU instance.

2) When there are concurrent IOPTE changes (followed by invalidations) and a domain attachment, the new attachment must not become out of sync at the HW level, meaning that an STE store and invalidation array load must be sequenced by the CPU's memory model.

3) When an ATS-enabled device attaches to a blocking domain, the core code requires a hard fence to ensure all ATS invalidations to the device are completed. Relying on RCU alone requires calling synchronize_rcu() that can be too slow. Instead, when ATS is in use, hold a conditional rwlock till all concurrent invalidations are finished.

Related future work and dependent projects:

* NVIDIA is building systems with > 10 SMMU instances where > 8 are being used concurrently in a single VM. So having 8 copies of an identical S2 page table is not efficient. Instead, all vSMMU instances should check compatibility on a shared S2 iopt, to eliminate 7 copies.

Previous attempt based on the list/spinlock design:
iommu/arm-smmu-v3: Allocate vmid per vsmmu instead of s2_parent
https://lore.kernel.org/all/[email protected]/
now can adopt this invs array, avoiding adding complex lists/locks.

* The guest support for BTM requires temporarily invalidating two ASIDs for a single instance. When it renumbers ASIDs this can now be done via the invs array.

* SVA with multiple devices being used by a single process (NVIDIA today has 4-8) sequentially iterates the invalidations through all instances. This ignores the HW concurrency available in each instance. It would be nice to not spin on each sync but go forward and issue batches to other instances also. Reducing to a single SVA domain shared across instances is required to look at this.

This is on Github:
https://github.com/nicolinc/iommufd/commits/arm_smmu_invs-rfcv2

Changelog
v2:
* Rebase v6.17-rc5
* Improve kdocs and inline comments
* Add arm_smmu_invs_dbg() for tracing
* Use users refcount to replace todel flag
* Initialize num_invs in arm_smmu_invs_alloc()
* Add a struct arm_smmu_inv_state to group invs pointers
* Add in struct arm_smmu_invs two flags (has_ats and old)
* Rename master->invs to master->build_invs, and sort the array
* Rework arm_smmu_domain_inv_range() and arm_smmu_invs_end_batch()
* Copy entries by struct arm_smmu_inv in arm_smmu_master_build_invs()
* Add arm_smmu_invs_flush_iotlb_tags() for IOTLB flush by last device
* Rework three invs mutation helpers, and prioritize use the in-place mutation for detach
* Take writer's lock unconditionally but keep it short, and only take reader's lock conditionally on a has_ats flag
v1:
https://lore.kernel.org/all/[email protected]/

Thanks
Nicolin

Jason Gunthorpe (1):
iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

Nicolin Chen (7):
iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free() iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range() iommu/arm-smmu-v3: Perform per-domain invalidations using
arm_smmu_invs

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   | 136 ++-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |  32 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  93 ++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 891 +++++++++++++++---
 4 files changed, 976 insertions(+), 176 deletions(-)
Signature · 1 line
2.43.0
#7 lore raw

RFC PATCH v2 2/8 iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA

Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB.

Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array.

There is no function change.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 3 +++
 3 files changed, 5 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -0
index ae23aacc38402..5c0b38595d209 100644
@@ -858,6 +858,7 @@ struct arm_smmu_master {
 enum arm_smmu_domain_stage {
 	ARM_SMMU_DOMAIN_S1 = 0,
 	ARM_SMMU_DOMAIN_S2,
+	ARM_SMMU_DOMAIN_SVA,
 };
 
 struct arm_smmu_domain {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -0
index 59a480974d80f..6097f1f540d87 100644
@@ -346,6 +346,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
 	 * ARM_SMMU_FEAT_RANGE_INV is present
 	 */
 	smmu_domain->domain.pgsize_bitmap = PAGE_SIZE;
+	smmu_domain->stage = ARM_SMMU_DOMAIN_SVA;
 	smmu_domain->smmu = smmu;
 
 	ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -0
index cccf8f52ee0d5..0016ec699acfe 100644
@@ -3070,6 +3070,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		arm_smmu_install_ste_for_dev(master, &target);
 		arm_smmu_clear_cd(master, IOMMU_NO_PASID);
 		break;
+	default:
+		WARN_ON(true);
+		break;
 	}
 
 	arm_smmu_attach_commit(&state);
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH v2 1/8 iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit

In reply to Nicolin Chen

None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a wrapper function, for the new arm_smmu_invs structure.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -2
index 2a8b46b948f05..cccf8f52ee0d5 100644
@@ -974,11 +974,17 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu,
 	cmds->num++;
 }
 
+/* Clears cmds->num after a successful submission */
 static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 				      struct arm_smmu_cmdq_batch *cmds)
 {
-	return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
-					   cmds->num, true);
+	int ret;
+
+	ret = arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
+					  cmds->num, true);
+	if (!ret)
+		cmds->num = 0;
+	return ret;
 }
 
 static void arm_smmu_page_response(struct device *dev, struct iopf_fault *unused,
Signature · 1 line
2.43.0
#20 lore raw

RFC PATCH v2 5/8 iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array

In reply to Nicolin Chen

When a master is attached from an old domain to a new domain, it needs to build an invalidation array to delete and add the array entries from/onto the invalidation arrays of those two domains, passed via the to_merge and to_unref arguments into arm_smmu_invs_merge/unref() respectively.

Since the master->num_streams might differ across masters, a memory would have to be allocated when building an to_merge/to_unref array which might fail with -ENOMEM.

On the other hand, an attachment to arm_smmu_blocked_domain must not fail so it's the best to avoid any memory allocation in that path.

Pre-allocate a fixed size invalidation array for every master. This array will be used as a scratch to fill dynamically when building a to_merge or to_unref invs array.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  8 +++++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 +++++++++++++++++++++
 2 files changed, 34 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8 -0
index 34fcc1a930e6a..246c6d84de3ab 100644
@@ -919,6 +919,14 @@ struct arm_smmu_master {
 	struct arm_smmu_device		*smmu;
 	struct device			*dev;
 	struct arm_smmu_stream		*streams;
+	/*
+	 * Scratch memory for a to_merge or to_unref array to build a per-domain
+	 * invalidation array. It'll be pre-allocated with enough enries for all
+	 * possible build scenarios. It can be used by only one caller at a time
+	 * until the arm_smmu_invs_merge/unref() finishes. Must be locked by the
+	 * iommu_group mutex.
+	 */
+	struct arm_smmu_invs		*build_invs;
 	struct arm_smmu_vmaster		*vmaster; /* use smmu->streams_mutex */
 	/* Locked by the iommu core using the group mutex */
 	struct arm_smmu_ctx_desc_cfg	cd_table;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +26 -0
index 83d842bd88817..4e69c81f5a28b 100644
@@ -3663,12 +3663,22 @@ static int arm_smmu_init_sid_strtab(struct arm_smmu_device *smmu, u32 sid)
 	return 0;
 }
 
+static int arm_smmu_ids_cmp(const void *_l, const void *_r)
+{
+	const typeof_member(struct iommu_fwspec, ids[0]) *l = _l;
+	const typeof_member(struct iommu_fwspec, ids[0]) *r = _r;
+
+	return cmp_int(*l, *r);
+}
+
 static int arm_smmu_insert_master(struct arm_smmu_device *smmu,
 				  struct arm_smmu_master *master)
 {
 	int i;
 	int ret = 0;
 	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev);
+	bool ats_supported = dev_is_pci(master->dev) &&
+			     pci_ats_supported(to_pci_dev(master->dev));
 
 	master->streams = kcalloc(fwspec->num_ids, sizeof(*master->streams),
 				  GFP_KERNEL);
@@ -3676,6 +3686,20 @@ static int arm_smmu_insert_master(struct arm_smmu_device *smmu,
 		return -ENOMEM;
 	master->num_streams = fwspec->num_ids;
 
+	/* Base case has 1 ASID or 1~2 VMIDs. ATS case adds num_ids */
+	if (!ats_supported)
+		master->build_invs = arm_smmu_invs_alloc(2);
+	else
+		master->build_invs = arm_smmu_invs_alloc(2 + fwspec->num_ids);
+	if (IS_ERR(master->build_invs)) {
+		kfree(master->streams);
+		return PTR_ERR(master->build_invs);
+	}
+
+	/* Put the ids into order for a sorted to_merge or to_unref array */
+	sort_nonatomic(fwspec->ids, fwspec->num_ids, sizeof(fwspec->ids[0]),
+		       arm_smmu_ids_cmp, NULL);
+
 	mutex_lock(&smmu->streams_mutex);
 	for (i = 0; i < fwspec->num_ids; i++) {
 		struct arm_smmu_stream *new_stream = &master->streams[i];
@@ -3713,6 +3737,7 @@ static int arm_smmu_insert_master(struct arm_smmu_device *smmu,
 		for (i--; i >= 0; i--)
 			rb_erase(&master->streams[i].node, &smmu->streams);
 		kfree(master->streams);
+		kfree(master->build_invs);
 	}
 	mutex_unlock(&smmu->streams_mutex);
 
@@ -3734,6 +3759,7 @@ static void arm_smmu_remove_master(struct arm_smmu_master *master)
 	mutex_unlock(&smmu->streams_mutex);
 
 	kfree(master->streams);
+	kfree(master->build_invs);
 }
 
 static struct iommu_device *arm_smmu_probe_device(struct device *dev)
Signature · 1 line
2.43.0
#14 lore raw

RFC PATCH v2 4/8 iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

In reply to Nicolin Chen

From: Jason Gunthorpe <[email protected]>

Create a new data structure to hold an array of invalidations that need to be performed for the domain based on what masters are attached, to replace the single smmu pointer and linked list of masters in the current design.

Each array entry holds one of the invalidation actions - S1_ASID, S2_VMID, ATS or their variant with information to feed invalidation commands to HW. It is structured so that multiple SMMUs can participate in the same array, removing one key limitation of the current system.

To maximize performance, a sorted array is used as the data structure. It allows grouping SYNCs together to parallelize invalidations. For instance, it will group all the ATS entries after the ASID/VMID entry, so they will all be pushed to the PCI devices in parallel with one SYNC.

To minimize the locking cost on the invalidation fast path (reader of the invalidation array), the array is managed with RCU.

Provide a set of APIs to add/delete entries to/from an array, which cover cannot-fail attach cases, e.g. attaching to arm_smmu_blocked_domain. Also add kunit coverage for those APIs.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  79 +++++++
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  93 ++++++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 212 ++++++++++++++++++
 3 files changed, 384 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +79 -0
index 96a23ca633cb6..34fcc1a930e6a 100644
@@ -649,6 +649,82 @@ struct arm_smmu_cmdq_batch {
 	int				num;
 };
 
+/*
+ * The order here also determines the sequence in which commands are sent to the
+ * command queue. E.g. TLBI must be done before ATC_INV.
+ */
+enum arm_smmu_inv_type {
+	INV_TYPE_S1_ASID,
+	INV_TYPE_S2_VMID,
+	INV_TYPE_S2_VMID_S1_CLEAR,
+	INV_TYPE_ATS,
+	INV_TYPE_ATS_FULL,
+};
+
+struct arm_smmu_inv {
+	struct arm_smmu_device *smmu;
+	u8 type;
+	u8 size_opcode;
+	u8 nsize_opcode;
+	u32 id; /* ASID or VMID or SID */
+	union {
+		size_t pgsize; /* ARM_SMMU_FEAT_RANGE_INV */
+		u32 ssid; /* INV_TYPE_ATS */
+	};
+
+	refcount_t users; /* users=0 to mark as a trash to be purged */
+};
+
+/**
+ * struct arm_smmu_invs - Per-domain invalidation array
+ * @num_invs: number of invalidations in the flexible array
+ * @rcu: rcu head for kfree_rcu()
+ * @inv: flexible invalidation array
+ *
+ * The arm_smmu_invs is an RCU data structure. During a ->attach_dev callback,
+ * arm_smmu_invs_merge(), arm_smmu_invs_unref() and arm_smmu_invs_purge() will
+ * be used to allocate a new copy of an old array for addition and deletion in
+ * the old domain's and new domain's invs arrays.
+ *
+ * The arm_smmu_invs_unref() mutates a given array, by internally reducing the
+ * users counts of some given entries. This exists to support a no-fail routine
+ * like attaching to an IOMMU_DOMAIN_BLOCKED. And it could pair with a followup
+ * arm_smmu_invs_purge() call to generate a new clean array.
+ *
+ * Concurrent invalidation thread will push every invalidation described in the
+ * array into the command queue for each invalidation event. It is designed like
+ * this to optimize the invalidation fast path by avoiding locks.
+ *
+ * A domain can be shared across SMMU instances. When an instance gets removed,
+ * it would delete all the entries that belong to that SMMU instance. Then, a
+ * synchronize_rcu() would have to be called to sync the array, to prevent any
+ * concurrent invalidation thread accessing the old array from issuing commands
+ * to the command queue of a removed SMMU instance.
+ */
+struct arm_smmu_invs {
+	size_t num_invs;
+	struct rcu_head rcu;
+	struct arm_smmu_inv inv[];
+};
+
+static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
+{
+	struct arm_smmu_invs *new_invs;
+
+	new_invs = kzalloc(struct_size(new_invs, inv, num_invs), GFP_KERNEL);
+	if (!new_invs)
+		return ERR_PTR(-ENOMEM);
+	new_invs->num_invs = num_invs;
+	return new_invs;
+}
+
+struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
+					  struct arm_smmu_invs *to_merge);
+size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
+			   struct arm_smmu_invs *to_unref);
+struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
+					  size_t num_dels);
+
 struct arm_smmu_evtq {
 	struct arm_smmu_queue		q;
 	struct iopf_queue		*iopf;
@@ -875,6 +951,8 @@ struct arm_smmu_domain {
 
 	struct iommu_domain		domain;
 
+	struct arm_smmu_invs __rcu	*invs;
+
 	/* List of struct arm_smmu_master_domain */
 	struct list_head		devices;
 	spinlock_t			devices_lock;
@@ -956,6 +1034,7 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void);
 
 static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domain)
 {
+	kfree_rcu(smmu_domain->invs, rcu);
 	kfree(smmu_domain);
 }
 
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c +93 -0

105 lines not on this page · raw message

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +212 -0

237 lines not on this page · raw message

Signature · 1 line
2.43.0
#34 lore raw

RFC PATCH v2 8/8 iommu/arm-smmu-v3: Perform per-domain invalidations using arm_smmu_invs

In reply to Nicolin Chen

Replace the old invalidation functions with arm_smmu_domain_inv_range() in all the existing invalidation routines. And deprecate the old functions.

The new arm_smmu_domain_inv_range() handles the CMDQ_MAX_TLBI_OPS as well, so drop it in the SVA function.

Since arm_smmu_cmdq_batch_add_range() has only one caller now, and it must be given a valid size, add a WARN_ON_ONCE to catch any missed case.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   7 -
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |  29 +--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 165 +-----------------
 3 files changed, 11 insertions(+), 190 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -7
index c73a94514c6d6..e9f97301ded31 100644
@@ -1072,13 +1072,6 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		       struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
 		       struct arm_smmu_cd *cd, struct iommu_domain *old);
 
-void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
-void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
-				 size_t granule, bool leaf,
-				 struct arm_smmu_domain *smmu_domain);
-int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
-			    unsigned long iova, size_t size);
-
 void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
 			       unsigned long iova, size_t size,
 			       unsigned int granule, bool leaf);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +3 -26
index fc601b494e0af..048b53f79b144 100644
@@ -122,15 +122,6 @@ void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
 }
 EXPORT_SYMBOL_IF_KUNIT(arm_smmu_make_sva_cd);
 
-/*
- * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this
- * is used as a threshold to replace per-page TLBI commands to issue in the
- * command queue with an address-space TLBI command, when SMMU w/o a range
- * invalidation feature handles too many per-page TLBI commands, which will
- * otherwise result in a soft lockup.
- */
-#define CMDQ_MAX_TLBI_OPS		(1 << (PAGE_SHIFT - 3))
-
 static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
 						struct mm_struct *mm,
 						unsigned long start,
@@ -146,21 +137,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
 	 * range. So do a simple translation here by calculating size correctly.
 	 */
 	size = end - start;
-	if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_RANGE_INV)) {
-		if (size >= CMDQ_MAX_TLBI_OPS * PAGE_SIZE)
-			size = 0;
-	} else {
-		if (size == ULONG_MAX)
-			size = 0;
-	}
-
-	if (!size)
-		arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
-	else
-		arm_smmu_tlb_inv_range_asid(start, size, smmu_domain->cd.asid,
-					    PAGE_SIZE, false, smmu_domain);
 
-	arm_smmu_atc_inv_domain(smmu_domain, start, size);
+	arm_smmu_domain_inv_range(smmu_domain, start, size, PAGE_SIZE, false);
 }
 
 static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
@@ -191,8 +169,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
 	}
 	spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
 
-	arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
-	arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
+	arm_smmu_domain_inv(smmu_domain);
 }
 
 static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn)
@@ -301,7 +278,7 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
 	/*
 	 * Ensure the ASID is empty in the iommu cache before allowing reuse.
 	 */
-	arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
+	arm_smmu_domain_inv(smmu_domain);
 
 	/*
 	 * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -157

211 lines not on this page · raw message

Signature · 1 line
2.43.0
#31 lore raw

RFC PATCH v2 7/8 iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range()

In reply to Nicolin Chen

Now, each smmu_domain is built with an invs array that keeps all the IDs (ASID/VMID) and its attached device SIDs, following the exact pattern of all the existing invalidation functions.

Introduce a new arm_smmu_domain_inv helper iterating smmu_domain->invs, to convert the invalidation array to commands. Any invalidation request with no size specified means an entire flush over a range based one.

ATC invalidations must be in sync with an attachment, espeically when it is an IOMMU_DOMAIN_BLOCKED that is used by the core for a device reset. Add a has_ats flag in the invs structure and hold the read lock if it is set, in which way a concurrent attachment will be blocked until any ATC invalidation being sent to the in the command queue is finished.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  16 ++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 225 ++++++++++++++++++--
 2 files changed, 228 insertions(+), 13 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +16 -0
index e4e0e066108cc..c73a94514c6d6 100644
@@ -675,9 +675,15 @@ struct arm_smmu_inv {
 	refcount_t users; /* users=0 to mark as a trash to be purged */
 };
 
+static inline bool arm_smmu_inv_is_ats(struct arm_smmu_inv *inv)
+{
+	return inv->type == INV_TYPE_ATS || inv->type == INV_TYPE_ATS_FULL;
+}
+
 /**
  * struct arm_smmu_invs - Per-domain invalidation array
  * @num_invs: number of invalidations in the flexible array
+ * @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL
  * @old: flag to synchronize with reader
  * @rwlock: optional rwlock to fench ATS operations
  * @rcu: rcu head for kfree_rcu()
@@ -706,6 +712,7 @@ struct arm_smmu_inv {
 struct arm_smmu_invs {
 	size_t num_invs;
 	rwlock_t rwlock;
+	u8 has_ats;
 	u8 old;
 	struct rcu_head rcu;
 	struct arm_smmu_inv inv[];
@@ -1072,6 +1079,15 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
 int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
 			    unsigned long iova, size_t size);
 
+void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
+			       unsigned long iova, size_t size,
+			       unsigned int granule, bool leaf);
+
+static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain)
+{
+	arm_smmu_domain_inv_range(smmu_domain, 0, 0, 0, false);
+}
+
 void __arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu,
 			      struct arm_smmu_cmdq *cmdq);
 int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +212 -13

258 lines not on this page · raw message

Signature · 1 line
2.43.0
#24 lore raw

RFC PATCH v2 6/8 iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

In reply to Nicolin Chen

Update the invs array with the invalidations required by each domain type during attachment operations.

Only an SVA domain or a paging domain will have an invs array:
a. SVA domain will add an INV_TYPE_S1_ASID per SMMU and an INV_TYPE_ATS
per SID

b. Non-nesting-parent paging domain with no ATS-enabled master will add
a single INV_TYPE_S1_ASID or INV_TYPE_S2_VMID per SMMU

c. Non-nesting-parent paging domain with ATS-enabled master(s) will do
(b) and add an INV_TYPE_ATS per SID

d. Nesting-parent paging domain will add an INV_TYPE_S2_VMID followed by
an INV_TYPE_S2_VMID_S1_CLEAR per vSMMU. For an ATS-enabled master, it will add an INV_TYPE_ATS_FULL per SID

The per-domain invalidation is not needed, until the domain is attached to a master, i.e. a possible translation request. Giving this clears a way to allowing the domain to be attached to many SMMUs, and avoids any pointless invalidation overheads during a teardown if there are no STE/CDs referring to the domain. This also means, when the last device is detached, the old domain must flush its ASID or VMID because any iommu_unmap() call after it wouldn't initiate any invalidation given an empty domain invs array.

Introduce some arm_smmu_invs helper functions for building scratch arrays, preparing and installing old/new domain's invalidation arrays.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  22 ++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 312 +++++++++++++++++++-
 2 files changed, 332 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +22 -0
index 246c6d84de3ab..e4e0e066108cc 100644
@@ -678,6 +678,8 @@ struct arm_smmu_inv {
 /**
  * struct arm_smmu_invs - Per-domain invalidation array
  * @num_invs: number of invalidations in the flexible array
+ * @old: flag to synchronize with reader
+ * @rwlock: optional rwlock to fench ATS operations
  * @rcu: rcu head for kfree_rcu()
  * @inv: flexible invalidation array
  *
@@ -703,6 +705,8 @@ struct arm_smmu_inv {
  */
 struct arm_smmu_invs {
 	size_t num_invs;
+	rwlock_t rwlock;
+	u8 old;
 	struct rcu_head rcu;
 	struct arm_smmu_inv inv[];
 };
@@ -714,6 +718,7 @@ static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
 	new_invs = kzalloc(struct_size(new_invs, inv, num_invs), GFP_KERNEL);
 	if (!new_invs)
 		return ERR_PTR(-ENOMEM);
+	rwlock_init(&new_invs->rwlock);
 	new_invs->num_invs = num_invs;
 	return new_invs;
 }
@@ -1082,6 +1087,21 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)
 	       IOMMU_FWSPEC_PCI_RC_CANWBS;
 }
 
+/**
+ * struct arm_smmu_inv_state - Per-domain invalidation array state
+ * @invs_ptr: points to the domain->invs (unwinding nesting/etc.) or is NULL if
+ *            no change should be made
+ * @old_invs: the original invs array
+ * @new_invs: for new domain, this is the new invs array to update domin->invs;
+ *            for old domain, this is the master->build_invs to pass in as the
+ *            to_unref argument to an arm_smmu_invs_unref() call
+ */
+struct arm_smmu_inv_state {
+	struct arm_smmu_invs **invs_ptr;
+	struct arm_smmu_invs *old_invs;
+	struct arm_smmu_invs *new_invs;
+};
+
 struct arm_smmu_attach_state {
 	/* Inputs */
 	struct iommu_domain *old_domain;
@@ -1091,6 +1111,8 @@ struct arm_smmu_attach_state {
 	ioasid_t ssid;
 	/* Resulting state */
 	struct arm_smmu_vmaster *vmaster;
+	struct arm_smmu_inv_state old_domain_invst;
+	struct arm_smmu_inv_state new_domain_invst;
 	bool ats_enabled;
 };
 
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +310 -2

360 lines not on this page · raw message

Signature · 1 line
2.43.0
Acked-by on v2 1/8
#2 lore raw

Re: [PATCH rfcv2 1/8] iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit

In reply to Nicolin Chen
Quoted 15 lines · Nicolin Chen

None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a wrapper function, for the new arm_smmu_invs structure.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 2a8b46b948f05..cccf8f52ee0d5 100644

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, · line 980

Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
@@ -974,11 +974,17 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu,
 	cmds->num++;
 }
 
+/* Clears cmds->num after a successful submission */
 static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 				      struct arm_smmu_cmdq_batch *cmds)
 {

Nit: arm_smmu_cmdq_batch_submit_clear()?

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, · line 990

Quoted 12 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7 -2
-	return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
-					   cmds->num, true);
+	int ret;
+
+	ret = arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
+					  cmds->num, true);
+	if (!ret)
+		cmds->num = 0;
+	return ret;
 }
 
 static void arm_smmu_page_response(struct device *dev, struct iopf_fault *unused,

Balbir

Acked-by on v2 2/8
#8 lore raw

Re: [PATCH rfcv2 2/8] iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA

In reply to Nicolin Chen
Quoted 11 lines · Nicolin Chen

Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB.

Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array.

There is no function change.

#3 lore raw

Re: [PATCH rfcv2 1/8] iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit

In reply to Balbir Singh
Quoted 19 lines · Balbir Singh
Quoted 18 lines · Nicolin Chen

None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a wrapper function, for the new arm_smmu_invs structure.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 2a8b46b948f05..cccf8f52ee0d5 100644
@@ -974,11 +974,17 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu,
 	cmds->num++;
 }
 
Quoted 6 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, · line 980

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+/* Clears cmds->num after a successful submission */
 static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 				      struct arm_smmu_cmdq_batch *cmds)
 {

Nit: arm_smmu_cmdq_batch_submit_clear()?

Probably not. There is no particular point in highlighting it in the function name, as there is no use case wanting an uncleared version.

Thanks
Nicolin

#15 lore raw

Re: [PATCH rfcv2 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

In reply to Nicolin Chen
kernel test robot noticed the following build errors:

Hi Nicolin,

kernel test robot noticed the following build errors:

[auto build test ERROR on soc/for-next]
[also build test ERROR on linus/master v6.17-rc5 next-20250909] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Nicolin-Chen/iommu-arm-smmu-v3-Clear-cmds-num-after-arm_smmu_cmdq_batch_submit/20250909-073052
base:   https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git for-next
patch link:    https://lore.kernel.org/r/80310b98efa4bd7e95d7b3ca302f40d4d69e59c5.1757373449.git.nicolinc%40nvidia.com
patch subject: [PATCH rfcv2 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
config: arm64-randconfig-003-20250909 (https://download.01.org/0day-ci/archive/20250909/[email protected]/config)
compiler: aarch64-linux-gcc (GCC) 11.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250909/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1082:23: error: static declaration of 'arm_smmu_invs_merge' follows non-static declaration
    1082 | struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
         |                       ^~~~~~~~~~~~~~~~~~~
   In file included from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:34:
   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:721:23: note: previous declaration of 'arm_smmu_invs_merge' with type 'struct arm_smmu_invs *(struct arm_smmu_invs *, struct arm_smmu_invs *)'
     721 | struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
         |                       ^~~~~~~~~~~~~~~~~~~
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1165:8: error: static declaration of 'arm_smmu_invs_unref' follows non-static declaration
    1165 | size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
         |        ^~~~~~~~~~~~~~~~~~~
   In file included from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:34:
   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:723:8: note: previous declaration of 'arm_smmu_invs_unref' with type 'size_t(struct arm_smmu_invs *, struct arm_smmu_invs *)' {aka 'long unsigned int(struct arm_smmu_invs *, struct arm_smmu_invs *)'}
     723 | size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
         |        ^~~~~~~~~~~~~~~~~~~
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1215:23: error: static declaration of 'arm_smmu_invs_purge' follows non-static declaration

1215 | struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
In file included from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:34:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:725:23: note: previous declaration of 'arm_smmu_invs_purge' with type 'struct arm_smmu_invs *(struct arm_smmu_invs *, size_t)' {aka 'struct arm_smmu_invs *(struct arm_smmu_invs *, long unsigned int)'}
725 | struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1215:23: warning: 'arm_smmu_invs_purge' defined but not used [-Wunused-function]
1215 | struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1165:8: warning: 'arm_smmu_invs_unref' defined but not used [-Wunused-function]
1165 | size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1082:23: warning: 'arm_smmu_invs_merge' defined but not used [-Wunused-function]
1082 | struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~

vim +/arm_smmu_invs_merge +1082 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c

1063
1064 /**
1065 * arm_smmu_invs_merge() - Merge @to_merge into @invs and generate a new array
1066 * @invs: the base invalidation array
1067 * @to_merge: an array of invlidations to merge
1068 *
1069 * Return: a newly allocated array on success, or ERR_PTR
1070 *
1071 * This function must be locked and serialized with arm_smmu_invs_unref() and
1072 * arm_smmu_invs_purge(), but do not lockdep on any lock for KUNIT test.
1073 *
1074 * Either @invs or @to_merge must be sorted itself. This ensures the returned
1075 * array will be sorted as well.
1076 *
1077 * Caller is resposible for freeing the @invs and the returned new one.
1078 *
1079 * Entries marked as trash will be purged in the returned array.
1080 */
1081 VISIBLE_IF_KUNIT

> 1082	struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
  1083						  struct arm_smmu_invs *to_merge)
  1084	{
  1085		struct arm_smmu_invs *new_invs;
  1086		struct arm_smmu_inv *new;
  1087		size_t num_adds = 0;
  1088		size_t num_dels = 0;
  1089		size_t i, j;
  1090	
  1091		for (i = j = 0; i != invs->num_invs || j != to_merge->num_invs;) {
  1092			int cmp = arm_smmu_invs_merge_cmp(invs, i, to_merge, j);
  1093	
  1094			if (cmp < 0) {
  1095				/* no found in to_merge, leave alone but delete trash */
  1096				if (!refcount_read(&invs->inv[i].users))
  1097					num_dels++;
  1098				i++;
  1099			} else if (cmp == 0) {
  1100				/* same item */
  1101				i++;
  1102				j++;
  1103			} else {
  1104				/* unique to to_merge */
  1105				num_adds++;
  1106				j++;
  1107			}
  1108		}
  1109	
  1110		new_invs = arm_smmu_invs_alloc(invs->num_invs - num_dels + num_adds);
  1111		if (IS_ERR(new_invs))
  1112			return new_invs;
  1113	
  1114		new = new_invs->inv;
  1115		for (i = j = 0; i != invs->num_invs || j != to_merge->num_invs;) {
  1116			int cmp = arm_smmu_invs_merge_cmp(invs, i, to_merge, j);
  1117	
  1118			if (cmp <= 0 && !refcount_read(&invs->inv[i].users)) {
  1119				i++;
  1120				continue;
  1121			}
  1122	
  1123			if (cmp < 0) {
  1124				*new = invs->inv[i];
  1125				i++;
  1126			} else if (cmp == 0) {
  1127				*new = invs->inv[i];
  1128				refcount_inc(&new->users);
  1129				i++;
  1130				j++;
  1131			} else {
  1132				*new = to_merge->inv[j];
  1133				refcount_set(&new->users, 1);
  1134				j++;
  1135			}
  1136			new++;
  1137		}
  1138	
  1139		WARN_ON(new != new_invs->inv + new_invs->num_invs);
  1140	
  1141		return new_invs;
  1142	}
  1143	EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge);
  1144	
  1145	/**
  1146	 * arm_smmu_invs_unref() - Find in @invs for all entries in @to_unref, decrease
  1147	 *                         the user counts without deletions
  1148	 * @invs: the base invalidation array
  1149	 * @to_unref: an array of invlidations to decrease their user counts
  1150	 *
  1151	 * Return: the number of trash entries in the array, for arm_smmu_invs_purge()
  1152	 *
  1153	 * This function will not fail. Any entry with users=0 will be marked as trash.
  1154	 * All trash entries will remain in the @invs until being completely deleted by
  1155	 * the next arm_smmu_invs_merge() or an arm_smmu_invs_purge() function call.
  1156	 *
  1157	 * This function must be locked and serialized with arm_smmu_invs_merge() and
  1158	 * arm_smmu_invs_purge(), but do not lockdep on any lock for KUNIT test.
  1159	 *
  1160	 * Note that the @invs->num_invs will not be updated, even if the actual number
  1161	 * of invalidations are decreased. Readers should take the read lock to iterate
  1162	 * each entry and check its users counter until @inv->num_invs.
  1163	 */
  1164	VISIBLE_IF_KUNIT
> 1165	size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
  1166				   struct arm_smmu_invs *to_unref)
  1167	{
  1168		size_t num_dels = 0;
  1169		size_t i, j;
  1170	
  1171		for (i = j = 0; i != invs->num_invs || j != to_unref->num_invs;) {
  1172			int cmp;
  1173	
  1174			if (!refcount_read(&invs->inv[i].users)) {
  1175				num_dels++;
  1176				i++;
  1177				continue;
  1178			}
  1179	
  1180			cmp = arm_smmu_invs_merge_cmp(invs, i, to_unref, j);
  1181			if (cmp < 0) {
  1182				/* not found in to_unref, leave alone */
  1183				i++;
  1184			} else if (cmp == 0) {
  1185				/* same item */
  1186				if (refcount_dec_and_test(&invs->inv[i].users))
  1187					num_dels++;
  1188				i++;
  1189				j++;
  1190			} else {
  1191				/* item in to_unref is not in invs or already a trash */
  1192				WARN_ON(true);
  1193				j++;
  1194			}
  1195		}
  1196		return num_dels;
  1197	}
  1198	EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_unref);
  1199	
  1200	/**
  1201	 * arm_smmu_invs_purge() - Purge all the trash entries in the @invs
  1202	 * @invs: the base invalidation array
  1203	 * @num_dels: expected number of trash entries, typically the return value from
  1204	 *            a prior arm_smmu_invs_unref() call
  1205	 *
  1206	 * Return: a newly allocated array on success removing all the trash entries, or
  1207	 *         NULL on failure
  1208	 *
  1209	 * This function must be locked and serialized with arm_smmu_invs_merge() and
  1210	 * arm_smmu_invs_unref(), but do not lockdep on any lock for KUNIT test.
  1211	 *
  1212	 * Caller is resposible for freeing the @invs and the returned new one.
  1213	 */
  1214	VISIBLE_IF_KUNIT
> 1215	struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,

1216 size_t num_dels)
1217 {
1218 struct arm_smmu_invs *new_invs;
1219 size_t i, j;
1220
1221 if (WARN_ON(invs->num_invs < num_dels))
1222 return NULL;
1223
1224 new_invs = arm_smmu_invs_alloc(invs->num_invs - num_dels);
1225 if (IS_ERR(new_invs))
1226 return NULL;
1227
1228 for (i = j = 0; i != invs->num_invs; i++) {
1229 if (!refcount_read(&invs->inv[i].users))
1230 continue;
1231 new_invs->inv[j] = invs->inv[i];
1232 j++;
1233 }
1234
1235 WARN_ON(j != new_invs->num_invs);
1236 return new_invs;
1237 }
1238 EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_purge);
1239

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#9 lore raw

Re: [PATCH rfcv2 2/8] iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA

In reply to Nicolin Chen
Quoted 46 lines · Nicolin Chen

Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB.

Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array.

There is no function change.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 3 +++
 3 files changed, 5 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -0
index ae23aacc38402..5c0b38595d209 100644
@@ -858,6 +858,7 @@ struct arm_smmu_master {
 enum arm_smmu_domain_stage {
 	ARM_SMMU_DOMAIN_S1 = 0,
 	ARM_SMMU_DOMAIN_S2,
+	ARM_SMMU_DOMAIN_SVA,
 };
 
 struct arm_smmu_domain {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -0
index 59a480974d80f..6097f1f540d87 100644
@@ -346,6 +346,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
 	 * ARM_SMMU_FEAT_RANGE_INV is present
 	 */
 	smmu_domain->domain.pgsize_bitmap = PAGE_SIZE;
+	smmu_domain->stage = ARM_SMMU_DOMAIN_SVA;
 	smmu_domain->smmu = smmu;
 
 	ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index cccf8f52ee0d5..0016ec699acfe 100644

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 3074

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -0
@@ -3070,6 +3070,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		arm_smmu_install_ste_for_dev(master, &target);
 		arm_smmu_clear_cd(master, IOMMU_NO_PASID);
 		break;
+	default:
+		WARN_ON(true);

WARN_ONCE_ONCE() and shoudn't ret be set to -EINVAL?

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 3078

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+		break;
 	}
 
 	arm_smmu_attach_commit(&state);

Balbir

#4 lore raw

Re: [PATCH rfcv2 1/8] iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit

In reply to Nicolin Chen
Quoted 27 lines · Nicolin Chen
Quoted 19 lines · Balbir Singh
Quoted 18 lines · Nicolin Chen

None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a wrapper function, for the new arm_smmu_invs structure.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 2a8b46b948f05..cccf8f52ee0d5 100644
@@ -974,11 +974,17 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu,
 	cmds->num++;
 }
 
Quoted 6 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, · line 980

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+/* Clears cmds->num after a successful submission */
 static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 				      struct arm_smmu_cmdq_batch *cmds)
 {

Nit: arm_smmu_cmdq_batch_submit_clear()?

Quoted 3 lines

Probably not. There is no particular point in highlighting it in the function name, as there is no use case wanting an uncleared version.

I did not suggest we need an uncleared version, I suggested the change in name to highlight that the function has a side-effect of clearing the cmds->num

Thanks,
Balbir

#5 lore raw

Re: [PATCH rfcv2 1/8] iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit

Quoted 28 lines · Balbir Singh
Quoted 27 lines · Nicolin Chen
Quoted 25 lines · Balbir Singh
Quoted 15 lines · Nicolin Chen

None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a wrapper function, for the new arm_smmu_invs structure.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 2a8b46b948f05..cccf8f52ee0d5 100644

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, · line 980

Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
@@ -974,11 +974,17 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu,
 	cmds->num++;
 }
 
+/* Clears cmds->num after a successful submission */
 static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 				      struct arm_smmu_cmdq_batch *cmds)
 {

Nit: arm_smmu_cmdq_batch_submit_clear()?

Quoted 7 lines
Quoted 3 lines

Probably not. There is no particular point in highlighting it in the function name, as there is no use case wanting an uncleared version.

I did not suggest we need an uncleared version, I suggested the change in name to highlight that the function has a side-effect of clearing the cmds->num

No caller cares about the "side effect"...

Nicolin

#10 lore raw

Re: [PATCH rfcv2 2/8] iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA

In reply to Balbir Singh
Quoted 13 lines · Balbir Singh

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 3074

Quoted 10 lines · Nicolin Chen
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -0
index cccf8f52ee0d5..0016ec699acfe 100644
@@ -3070,6 +3070,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		arm_smmu_install_ste_for_dev(master, &target);
 		arm_smmu_clear_cd(master, IOMMU_NO_PASID);
 		break;
+	default:
+		WARN_ON(true);

WARN_ONCE_ONCE() and shoudn't ret be set to -EINVAL?

I will fix it.

Thanks
Nicolin

#6 lore raw

Re: [PATCH rfcv2 1/8] iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit

In reply to Nicolin Chen
Quoted 4 lines · Nicolin Chen

None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a wrapper function, for the new arm_smmu_invs structure.

This was added when arm_smmu_cmdq_batch_submit() was needed by a new arm_smmu_invs helper that used it at a few places. But after a few rounds of rework, I found that there is only one call now.

I will drop this patch.

Thanks
Nicolin

#16 lore raw

Re: [PATCH rfcv2 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

In reply to Nicolin Chen
Quoted 24 lines · Nicolin Chen
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +24 -0
+/**
+ * arm_smmu_invs_unref() - Find in @invs for all entries in @to_unref, decrease
+ *                         the user counts without deletions
+ * @invs: the base invalidation array
+ * @to_unref: an array of invlidations to decrease their user counts
+ *
+ * Return: the number of trash entries in the array, for arm_smmu_invs_purge()
+ *
+ * This function will not fail. Any entry with users=0 will be marked as trash.
+ * All trash entries will remain in the @invs until being completely deleted by
+ * the next arm_smmu_invs_merge() or an arm_smmu_invs_purge() function call.
+ *
+ * This function must be locked and serialized with arm_smmu_invs_merge() and
+ * arm_smmu_invs_purge(), but do not lockdep on any lock for KUNIT test.
+ *
+ * Note that the @invs->num_invs will not be updated, even if the actual number
+ * of invalidations are decreased. Readers should take the read lock to iterate
+ * each entry and check its users counter until @inv->num_invs.
+ */
+VISIBLE_IF_KUNIT
+size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
+			   struct arm_smmu_invs *to_unref)
+{
+	size_t num_dels = 0;

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) · line 1178

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10 -0
+	size_t i, j;
+
+	for (i = j = 0; i != invs->num_invs || j != to_unref->num_invs;) {
+		int cmp;
+
+		if (!refcount_read(&invs->inv[i].users)) {
+			num_dels++;
+			i++;
+			continue;
+		}

Found that this should AND the condition "i < invs->num_invs".

Will fix in next version.

Quoted 9 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +9 -0
+
+		cmp = arm_smmu_invs_merge_cmp(invs, i, to_unref, j);
+		if (cmp < 0) {
+			/* not found in to_unref, leave alone */
+			i++;
+		} else if (cmp == 0) {
+			/* same item */
+			if (refcount_dec_and_test(&invs->inv[i].users))
+				num_dels++;

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) · line 1197

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10 -0
+			i++;
+			j++;
+		} else {
+			/* item in to_unref is not in invs or already a trash */
+			WARN_ON(true);
+			j++;
+		}
+	}
+	return num_dels;
+}

Nicolin

Reviewed-by on v2 2/8
#11 lore raw

Re: [PATCH rfcv2 2/8] iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA

In reply to Nicolin Chen
Quoted 9 lines · Nicolin Chen

Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB.

Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array.

There is no function change.

Quoted 7 lines
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 3 +++
 3 files changed, 5 insertions(+)

Jason

Reviewed-by on v2 3/8
#13 lore raw

Re: [PATCH rfcv2 3/8] iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free()

In reply to Nicolin Chen
Quoted 10 lines · Nicolin Chen

There will be a bit more things to free than smmu_domain itself. So keep a simple inline function in the header to share aross files.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 5 +++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 4 ++--
 3 files changed, 8 insertions(+), 3 deletions(-)

Jason

#17 lore raw

Re: [PATCH rfcv2 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

In reply to Nicolin Chen
Quoted 11 lines · Nicolin Chen

+/**
+ * arm_smmu_invs_merge() - Merge @to_merge into @invs and generate a new array
+ * @invs: the base invalidation array
+ * @to_merge: an array of invlidations to merge
+ *
+ * Return: a newly allocated array on success, or ERR_PTR
+ *
+ * This function must be locked and serialized with arm_smmu_invs_unref() and
+ * arm_smmu_invs_purge(), but do not lockdep on any lock for KUNIT test.
+ *
+ * Either @invs or @to_merge must be sorted itself. This ensures the returned

s/Either/Both

A merge sort like this requires both lists to be sorted.

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
+struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
+					  struct arm_smmu_invs *to_merge)
+{
+	struct arm_smmu_invs *new_invs;

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) · line 1095

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10 -0
+	struct arm_smmu_inv *new;
+	size_t num_adds = 0;
+	size_t num_dels = 0;
+	size_t i, j;
+
+	for (i = j = 0; i != invs->num_invs || j != to_merge->num_invs;) {
+		int cmp = arm_smmu_invs_merge_cmp(invs, i, to_merge, j);
+
+		if (cmp < 0) {
+			/* no found in to_merge, leave alone but delete trash */

s/no/not/

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) · line 1098

Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -0
+			if (!refcount_read(&invs->inv[i].users))
+				num_dels++;
+			i++;

This sequence related to users should be consistent in all the merge sorts. The one below in unref is the best one:

+ int cmp;
+
+ if (!refcount_read(&invs->inv[i].users)) {
+ num_dels++;
+ i++;
+ continue;
+ }
+
+ cmp = arm_smmu_invs_merge_cmp(invs, i, to_unref, j);

Make all of these loops look like that

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) · line 1141

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
+
+	WARN_ON(new != new_invs->inv + new_invs->num_invs);
+
+	return new_invs;

A debugging check that the output list is sorted would be a nice touch for robustness.

I think this looks OK and has turned out to be pretty simple.

I've been thinking about generalizing it to core code and I think it would hold up well there as well?

Jason

Reviewed-by on v2 5/8
#21 lore raw

Re: [PATCH rfcv2 5/8] iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array

In reply to Nicolin Chen
Quoted 9 lines · Nicolin Chen

When a master is attached from an old domain to a new domain, it needs to build an invalidation array to delete and add the array entries from/onto the invalidation arrays of those two domains, passed via the to_merge and to_unref arguments into arm_smmu_invs_merge/unref() respectively.

Since the master->num_streams might differ across masters, a memory would have to be allocated when building an to_merge/to_unref array which might fail with -ENOMEM.

Quoted 6 lines

On the other hand, an attachment to arm_smmu_blocked_domain must not fail so it's the best to avoid any memory allocation in that path.

Pre-allocate a fixed size invalidation array for every master. This array will be used as a scratch to fill dynamically when building a to_merge or to_unref invs array.

Mention the sorting here too, though maybe that hunk should be moved to the next patch.

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_insert_master(struct arm_smmu_device *smmu, · line 3701

Quoted 13 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +13 -0
+	/* Base case has 1 ASID or 1~2 VMIDs. ATS case adds num_ids */
+	if (!ats_supported)
+		master->build_invs = arm_smmu_invs_alloc(2);
+	else
+		master->build_invs = arm_smmu_invs_alloc(2 + fwspec->num_ids);
+	if (IS_ERR(master->build_invs)) {
+		kfree(master->streams);
+		return PTR_ERR(master->build_invs);
+	}
+
+	/* Put the ids into order for a sorted to_merge or to_unref array */
+	sort_nonatomic(fwspec->ids, fwspec->num_ids, sizeof(fwspec->ids[0]),
+		       arm_smmu_ids_cmp, NULL);

The sort could be moved under the above !ats_supported, a little more insurance in case something is inspecting the ids.

I searched around and only found tegra_dev_iommu_get_stream_id() which is fine with the sort due to num_ids ==1

Otherwise looks OK

Jason

#25 lore raw

Re: [PATCH rfcv2 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

In reply to Nicolin Chen
Quoted 10 lines · Nicolin Chen

Update the invs array with the invalidations required by each domain type during attachment operations.

Only an SVA domain or a paging domain will have an invs array:
a. SVA domain will add an INV_TYPE_S1_ASID per SMMU and an INV_TYPE_ATS
per SID

b. Non-nesting-parent paging domain with no ATS-enabled master will add
a single INV_TYPE_S1_ASID or INV_TYPE_S2_VMID per SMMU

Quoted 6 lines

c. Non-nesting-parent paging domain with ATS-enabled master(s) will do
(b) and add an INV_TYPE_ATS per SID

d. Nesting-parent paging domain will add an INV_TYPE_S2_VMID followed by
an INV_TYPE_S2_VMID_S1_CLEAR per vSMMU. For an ATS-enabled master, it will add an INV_TYPE_ATS_FULL per SID

Just some minor forward looking clarification - this behavior should be triggered when a nest-parent is attached through the viommu using a nesting domain with a vSTE.

A nesting-parent that is just normally attached should act like a normal S2 since it does not and can not have a two stage S1 on top of it.

We can't quite get there yet until the next series of changing how the VMID allocation works.

Quoted 7 lines

The per-domain invalidation is not needed, until the domain is attached to a master, i.e. a possible translation request. Giving this clears a way to allowing the domain to be attached to many SMMUs, and avoids any pointless invalidation overheads during a teardown if there are no STE/CDs referring to the domain. This also means, when the last device is detached, the old domain must flush its ASID or VMID because any iommu_unmap() call after it wouldn't initiate any invalidation given an empty domain invs array.

Grammar/phrasing in this paragraph

Quoted 34 lines

Introduce some arm_smmu_invs helper functions for building scratch arrays, preparing and installing old/new domain's invalidation arrays.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  22 ++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 312 +++++++++++++++++++-
 2 files changed, 332 insertions(+), 2 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4 -0
index 246c6d84de3ab..e4e0e066108cc 100644
@@ -678,6 +678,8 @@ struct arm_smmu_inv {
 /**
  * struct arm_smmu_invs - Per-domain invalidation array
  * @num_invs: number of invalidations in the flexible array
+ * @old: flag to synchronize with reader
+ * @rwlock: optional rwlock to fench ATS operations
  * @rcu: rcu head for kfree_rcu()
  * @inv: flexible invalidation array
  *
@@ -703,6 +705,8 @@ struct arm_smmu_inv {
  */
 struct arm_smmu_invs {
 	size_t num_invs;
+	rwlock_t rwlock;
+	u8 old;
 	struct rcu_head rcu;
 	struct arm_smmu_inv inv[];
 };

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs) · line 724

Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -0
@@ -714,6 +718,7 @@ static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
 	new_invs = kzalloc(struct_size(new_invs, inv, num_invs), GFP_KERNEL);
 	if (!new_invs)
 		return ERR_PTR(-ENOMEM);
+	rwlock_init(&new_invs->rwlock);
 	new_invs->num_invs = num_invs;
 	return new_invs;
 }

Put these and related hunks in the patch adding arm_smmu_invs

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs, · line 1189

Quoted 9 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3 -1
@@ -1183,8 +1183,11 @@ size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
 			i++;
 		} else if (cmp == 0) {
 			/* same item */
-			if (refcount_dec_and_test(&invs->inv[i].users))
+			if (refcount_dec_and_test(&invs->inv[i].users)) {
+				/* Notify the caller about this deletion */
+				refcount_set(&to_unref->inv[j].users, 1);
 				num_dels++;

This is a bit convoluted. Instead of marking the entry and then iterating the list again just directly call a function to do the invalidation.

Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6 -0
+static inline void arm_smmu_invs_dbg(struct arm_smmu_master *master,
+				     struct arm_smmu_domain *smmu_domain,
+				     struct arm_smmu_invs *invs, char *name)
+{
+	size_t i;
+

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +10 -0
+	dev_dbg(master->dev, "domain (type: %x), invs: %s, num_invs: %ld\n",
+		smmu_domain->domain.type, name, invs->num_invs);
+	for (i = 0; i < invs->num_invs; i++) {
+		struct arm_smmu_inv *cur = &invs->inv[i];
+
+		dev_dbg(master->dev,
+			"  entry: inv[%ld], type: %u, id: %u, users: %u\n", i,
+			cur->type, cur->id, refcount_read(&cur->users));
+	}
+}

Move all the debug code to its own commit and don't send it

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4 -0
+static void
+arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state)
+{
+	struct arm_smmu_inv_state *invst = &state->new_domain_invst;

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +10 -0
+
+	if (!invst->invs_ptr)
+		return;
+
+	rcu_assign_pointer(*invst->invs_ptr, invst->new_invs);
+	/*
+	 * Committed to updating the STE, using the new invalidation array, and
+	 * acquiring any racing IOPTE updates.
+	 */
+	smp_mb();

We are commited to updating the STE. Make the invalidation list visable to parallel map/unmap threads and acquire any racying IOPTE updates.

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h

Quoted 13 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +13 -0
+	kfree_rcu(invst->old_invs, rcu);
+}
+
+/*
+ * When an array entry's users count reaches zero, it means the ASID/VMID is no
+ * longer being invalidated by map/unmap and must be cleaned. The rule is that
+ * all ASIDs/VMIDs not in an invalidation array are left cleared in the IOTLB.
+ */
+static void arm_smmu_invs_flush_iotlb_tags(struct arm_smmu_invs *invs)
+{
+	size_t i;
+
+	for (i = 0; i != invs->num_invs; i++) {

Just remove the loop and accept struct arm_smmu_inv * and call it directly.

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h

Quoted 12 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +12 -0
+	if (!new_invs) {
+		size_t new_num = old_invs->num_invs;
+
+		/*
+		 * OOM. Couldn't make a copy. Leave the array unoptimized. But
+		 * trim its size if some tailing entries are marked as trash.
+		 */
+		while (new_num != 0) {
+			if (refcount_read(&old_invs->inv[new_num - 1].users))
+				break;
+			new_num--;
+		}

Would be nicer to have arm_smmu_invs_unref return the new size so we don't need this loop

Looks Ok to me otherwise

Jason

#32 lore raw

Re: [PATCH rfcv2 7/8] iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range()

In reply to Nicolin Chen

On Mon, Sep 08, 2025 at 04:27:01PM -0700, Nicolin Chen wrote:

Maybe:

Each smmu_domain now has an arm_smmu_invs that specifies the invalidation steps to perform after any change the ioptes. This includes basic ASID/VMID, the special case for nesting, and ATS.

Introduce a new arm_smmu_domain_inv helper iterating smmu_domain->invs, to convert the invalidation array to commands. Any invalidation request with no size specified means an entire flush over a range based one.

Take advantage of the sorted list to compatible batch same-SMMU operations together. For instance ATS invaliations for multiple SIDs can be pushed as a batch.

ATS invalidations must be completed before the driver disables ATS or the device is permitted to ignore the racing invalidation and cause a SMMU timeout. This sequencing is done with a rwlock where holding the write side of the rwlock means there are no outstanding ATS invalidations. If ATS is not used the rwlock is ignored, similar to the existing code.

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_inv { · line 681

Quoted 12 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4 -0
index e4e0e066108cc..c73a94514c6d6 100644
@@ -675,9 +675,15 @@ struct arm_smmu_inv {
 	refcount_t users; /* users=0 to mark as a trash to be purged */
 };
 
+static inline bool arm_smmu_inv_is_ats(struct arm_smmu_inv *inv)
+{
+	return inv->type == INV_TYPE_ATS || inv->type == INV_TYPE_ATS_FULL;
+}

I would put these has_ats related infrastructure hunks in the first patch adding arm_smmu_invs

Jason

#18 lore raw

Re: [PATCH rfcv2 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

In reply to Jason Gunthorpe
Quoted 8 lines · Jason Gunthorpe

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) · line 1141

Quoted 4 lines · Nicolin Chen
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
+
+	WARN_ON(new != new_invs->inv + new_invs->num_invs);
+
+	return new_invs;

A debugging check that the output list is sorted would be a nice touch for robustness.

Sure. The second for loop generating "new" could run a sanity.

Quoted 4 lines

I think this looks OK and has turned out to be pretty simple.

I've been thinking about generalizing it to core code and I think it would hold up well there as well?

How would you like it to be written in the core: a generalized structure or a general macro that generates driver structures?

And potentially we need a new drivers/iommu/iommu-invs.c?

Thanks
Nicolin

#22 lore raw

Re: [PATCH rfcv2 5/8] iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array

In reply to Jason Gunthorpe
Quoted 17 lines · Jason Gunthorpe

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static int arm_smmu_insert_master(struct arm_smmu_device *smmu, · line 3701

Quoted 13 lines · Nicolin Chen
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +13 -0
+	/* Base case has 1 ASID or 1~2 VMIDs. ATS case adds num_ids */
+	if (!ats_supported)
+		master->build_invs = arm_smmu_invs_alloc(2);
+	else
+		master->build_invs = arm_smmu_invs_alloc(2 + fwspec->num_ids);
+	if (IS_ERR(master->build_invs)) {
+		kfree(master->streams);
+		return PTR_ERR(master->build_invs);
+	}
+
+	/* Put the ids into order for a sorted to_merge or to_unref array */
+	sort_nonatomic(fwspec->ids, fwspec->num_ids, sizeof(fwspec->ids[0]),
+		       arm_smmu_ids_cmp, NULL);

The sort could be moved under the above !ats_supported, a little more insurance in case something is inspecting the ids.

You mean this:
----------------------------------------------------------------
@@ -4080,19 +4080,19 @@ static int arm_smmu_insert_master(struct arm_smmu_device *smmu,

        master->num_streams = fwspec->num_ids;

        /* Base case has 1 ASID or 1~2 VMIDs. ATS case adds num_ids */
-       if (!ats_supported)
+       if (!ats_supported) {
                master->build_invs = arm_smmu_invs_alloc(2);

- else
+ } else {
+ /* Put the ids into order for a sorted to_merge or to_unref array */
+ sort_nonatomic(fwspec->ids, fwspec->num_ids, sizeof(fwspec->ids[0]),

+                              arm_smmu_ids_cmp, NULL);
                master->build_invs = arm_smmu_invs_alloc(2 + fwspec->num_ids);
+       }
        if (IS_ERR(master->build_invs)) {
                kfree(master->streams);
                return PTR_ERR(master->build_invs);
        }

- /* Put the ids into order for a sorted to_merge or to_unref array */
- sort_nonatomic(fwspec->ids, fwspec->num_ids, sizeof(fwspec->ids[0]),
- arm_smmu_ids_cmp, NULL);
-

        mutex_lock(&smmu->streams_mutex);
        for (i = 0; i < fwspec->num_ids; i++) {
                struct arm_smmu_stream *new_stream = &master->streams[i];

----------------------------------------------------------------
?

Hmm, I am not sure how it insures against anything concurrent.

Maybe we should sort it in arm_smmu_of_xlate() each time when adding a new ID? Or iommu_fwspec_add_ids() itself could sort, since we are thinking of generalizing this array in the core?

Thanks
Nicolin

#26 lore raw

Re: [PATCH rfcv2 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

In reply to Jason Gunthorpe
Quoted 22 lines · Jason Gunthorpe
Quoted 10 lines · Nicolin Chen

Update the invs array with the invalidations required by each domain type during attachment operations.

Only an SVA domain or a paging domain will have an invs array:
a. SVA domain will add an INV_TYPE_S1_ASID per SMMU and an INV_TYPE_ATS
per SID

b. Non-nesting-parent paging domain with no ATS-enabled master will add
a single INV_TYPE_S1_ASID or INV_TYPE_S2_VMID per SMMU

Quoted 6 lines

c. Non-nesting-parent paging domain with ATS-enabled master(s) will do
(b) and add an INV_TYPE_ATS per SID

d. Nesting-parent paging domain will add an INV_TYPE_S2_VMID followed by
an INV_TYPE_S2_VMID_S1_CLEAR per vSMMU. For an ATS-enabled master, it will add an INV_TYPE_ATS_FULL per SID

Just some minor forward looking clarification - this behavior should be triggered when a nest-parent is attached through the viommu using a nesting domain with a vSTE.

Quoted 6 lines

A nesting-parent that is just normally attached should act like a normal S2 since it does not and can not have a two stage S1 on top of it.

We can't quite get there yet until the next series of changing how the VMID allocation works.

Yea, you are right. Let's add this:

Note that case #d is for the case when a nesting parent domain is attached through a vSMMU instance using a nested domain carrying a vSTE. This model will allocate a VMID per vSMMU instance v.s. the current driver allocating per S2 domain. So, this requires a few more patches for S2 domain sharing.

Quoted 9 lines
Quoted 7 lines

The per-domain invalidation is not needed, until the domain is attached to a master, i.e. a possible translation request. Giving this clears a way to allowing the domain to be attached to many SMMUs, and avoids any pointless invalidation overheads during a teardown if there are no STE/CDs referring to the domain. This also means, when the last device is detached, the old domain must flush its ASID or VMID because any iommu_unmap() call after it wouldn't initiate any invalidation given an empty domain invs array.

Grammar/phrasing in this paragraph

OK. I asked AI to rewrite it:

The per-domain invalidation is not needed until the domain is attached to a master (when it starts to possibly use TLB). This will make it possible to attach the domain to multiple SMMUs and avoid unnecessary invalidation overhead during teardown if no STEs/CDs refer to the domain. It also means that when the last device is detached, the old domain must flush its ASID or VMID, since any new iommu_unmap() call would not trigger invalidations given an empty domain->invs array.

Quoted 5 lines
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -1
@@ -1183,8 +1183,11 @@ size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
 			i++;
 		} else if (cmp == 0) {
 			/* same item */
-			if (refcount_dec_and_test(&invs->inv[i].users))
Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs, · line 1189

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -0
+			if (refcount_dec_and_test(&invs->inv[i].users)) {
+				/* Notify the caller about this deletion */
+				refcount_set(&to_unref->inv[j].users, 1);
 				num_dels++;

This is a bit convoluted. Instead of marking the entry and then iterating the list again just directly call a function to do the invalidation.

OK. If we want to generalize this arm_smmu_invs_unref function, I suppose we will need to pass in a callback function pointer.

Quoted 7 lines
Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7 -0
+	if (!new_invs) {
+		size_t new_num = old_invs->num_invs;
+
+		/*
+		 * OOM. Couldn't make a copy. Leave the array unoptimized. But
+		 * trim its size if some tailing entries are marked as trash.
+		 */
Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, · line 3334

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +5 -0
+		while (new_num != 0) {
+			if (refcount_read(&old_invs->inv[new_num - 1].users))
+				break;
+			new_num--;
+		}

Would be nicer to have arm_smmu_invs_unref return the new size so we don't need this loop

The "new size" must be invs->num_invs subtracting the number of the tailing trash entries. So, arm_smmu_invs_unref() would have to have the same loop validating the tailing entries, right?

(I will address all other comments as well)

Thanks!
Nicolin

#33 lore raw

Re: [PATCH rfcv2 7/8] iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range()

In reply to Jason Gunthorpe
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Nicolin Chen
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -0
index e4e0e066108cc..c73a94514c6d6 100644
@@ -675,9 +675,15 @@ struct arm_smmu_inv {
 	refcount_t users; /* users=0 to mark as a trash to be purged */
 };
 
Quoted 7 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_inv { · line 681

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4 -0
+static inline bool arm_smmu_inv_is_ats(struct arm_smmu_inv *inv)
+{
+	return inv->type == INV_TYPE_ATS || inv->type == INV_TYPE_ATS_FULL;
+}

I would put these has_ats related infrastructure hunks in the first patch adding arm_smmu_invs

OK. I was trying to move this (and the chunks in the other thread) closer to where they are being used for a cleaner review.

Thanks
Nicolin

#23 lore raw

Re: [PATCH rfcv2 5/8] iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array

In reply to Nicolin Chen
Quoted 1 line · Nicolin Chen

Hmm, I am not sure how it insures against anything concurrent.

Yes, not concurrent, just if someone is later looking at the IDs list and expecting it to be in a certain order. I could not find such a thing so I think it is safe.

Jason

#27 lore raw

Re: [PATCH rfcv2 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

In reply to Nicolin Chen
Quoted 16 lines · Nicolin Chen
Quoted 7 lines
Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7 -0
+	if (!new_invs) {
+		size_t new_num = old_invs->num_invs;
+
+		/*
+		 * OOM. Couldn't make a copy. Leave the array unoptimized. But
+		 * trim its size if some tailing entries are marked as trash.
+		 */
Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, · line 3334

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +5 -0
+		while (new_num != 0) {
+			if (refcount_read(&old_invs->inv[new_num - 1].users))
+				break;
+			new_num--;
+		}

Would be nicer to have arm_smmu_invs_unref return the new size so we don't need this loop

Quoted 3 lines

The "new size" must be invs->num_invs subtracting the number of the tailing trash entries. So, arm_smmu_invs_unref() would have to have the same loop validating the tailing entries, right?

It doesn't need another loop, it just need to record the index of the last valid entry while it is doing its own loop. If it reaches invs->num_invs then that will be the new length.

Jason

#19 lore raw

Re: [PATCH rfcv2 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

In reply to Nicolin Chen
Quoted 5 lines · Nicolin Chen
Quoted 2 lines

I've been thinking about generalizing it to core code and I think it would hold up well there as well?

How would you like it to be written in the core: a generalized structure or a general macro that generates driver structures?

I'm thinking to leave this here for now and revisit it with another driver someday.

Jason

#28 lore raw

Re: [PATCH rfcv2 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

In reply to Jason Gunthorpe
Quoted 18 lines · Jason Gunthorpe
Quoted 16 lines · Nicolin Chen
Quoted 15 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, · line 3334

Quoted 12 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +12 -0
+	if (!new_invs) {
+		size_t new_num = old_invs->num_invs;
+
+		/*
+		 * OOM. Couldn't make a copy. Leave the array unoptimized. But
+		 * trim its size if some tailing entries are marked as trash.
+		 */
+		while (new_num != 0) {
+			if (refcount_read(&old_invs->inv[new_num - 1].users))
+				break;
+			new_num--;
+		}

Would be nicer to have arm_smmu_invs_unref return the new size so we don't need this loop

Quoted 7 lines
Quoted 3 lines

The "new size" must be invs->num_invs subtracting the number of the tailing trash entries. So, arm_smmu_invs_unref() would have to have the same loop validating the tailing entries, right?

It doesn't need another loop, it just need to record the index of the last valid entry while it is doing its own loop. If it reaches invs->num_invs then that will be the new length.

arm_smmu_invs_purge() needs num_dels, while its fallback routine needs num_invs (new size). This forces arm_smmu_invs_unref() to return two numbers.

I see a cleaner way of handling this is to update invs->num_invs inside arm_smmu_invs_unref():
----------------------------------------------------------------
@@ -1209,6 +1216,13 @@ size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,

                        j++;
                }
        }
+
+       /* The lock is required to fence concurrent ATS operations. */
+       write_lock_irqsave(&invs->rwlock, flags);
+       /* Trim the size by removing tailing trash entries */
+       WRITE_ONCE(invs->num_invs, num_invs);
+       write_unlock_irqrestore(&invs->rwlock, flags);
+
        return num_dels;

}
EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_unref);
----------------------------------------------------------------

So the caller would look like:
----------------------------------------------------------------

	num_dels = arm_smmu_invs_unref(old_invs, invst->new_invs);
	if (!num_dels)
		return;
 
	new_invs = arm_smmu_invs_purge(old_invs, num_dels);
	if (!new_invs)
		return;

----------------------------------------------------------------

Thanks
Nicolin

#29 lore raw

Re: [PATCH rfcv2 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

Review size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs, · line 1224

Quoted 13 lines · Nicolin Chen

I see a cleaner way of handling this is to update invs->num_invs inside arm_smmu_invs_unref():
----------------------------------------------------------------

(quoted diff) +6 -0
@@ -1209,6 +1216,13 @@ size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
                        j++;
                }
        }
+
+       /* The lock is required to fence concurrent ATS operations. */
+       write_lock_irqsave(&invs->rwlock, flags);
+       /* Trim the size by removing tailing trash entries */
+       WRITE_ONCE(invs->num_invs, num_invs);
+       write_unlock_irqrestore(&invs->rwlock, flags);

That seems Ok

It means the arm_smmu_invs_unref() becomes the fence that guarentees the ATS is stopped for anything marked as trash.

Then the next steps can just be normal RCU and don't need rwlocking.

Jason

#30 lore raw

Re: [PATCH rfcv2 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters

Quoted 17 lines · Jason Gunthorpe

Review size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs, · line 1224

Quoted 13 lines · Nicolin Chen

I see a cleaner way of handling this is to update invs->num_invs inside arm_smmu_invs_unref():
----------------------------------------------------------------

(quoted diff) +6 -0
@@ -1209,6 +1216,13 @@ size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
                        j++;
                }
        }
+
+       /* The lock is required to fence concurrent ATS operations. */
+       write_lock_irqsave(&invs->rwlock, flags);
+       /* Trim the size by removing tailing trash entries */
+       WRITE_ONCE(invs->num_invs, num_invs);
+       write_unlock_irqrestore(&invs->rwlock, flags);

That seems Ok

Quoted 4 lines

It means the arm_smmu_invs_unref() becomes the fence that guarentees the ATS is stopped for anything marked as trash.

Then the next steps can just be normal RCU and don't need rwlocking.

Yea. The "old" flag could be dropped too:

        while (true) {
                invs = rcu_dereference(smmu_domain->invs);

                /*
                 * Avoid locking unless ATS is being used. No ATS invalidate can
                 * be going on after a domain is detached.
                 */
                locked = false;
-               if (invs->has_ats || READ_ONCE(invs->old)) {
+               if (invs->has_ats) {
                        read_lock(&invs->rwlock);

- if (invs->old) {
- read_unlock(&invs->rwlock);
- continue;
- }

                        locked = true;
                }
                break;
        }

Thanks
Nicolin