iommu/arm-smmu-v3: Introduce an RCU-protected invalidation array
Subsystems (3)
v3 cover iommu/arm-smmu-v3: Introduce an RCU-protected invalidation array
This is a work based on Jason's design and algorithm. This implementation follows his initial draft and revising as well.
The new arm_smmu_invs array is an RCU-protected array, mutated when device attach to the domain, iterated when an invalidation is required for IOPTE changes in this domain. This keeps the current invalidation efficiency of a smb_mb() followed by a conditional rwlock replacing the atomic/spinlock combination.
A new data structure is defined for the array and its entry, representing invalidation operations, such as S1_ASID, S2_VMID, and ATS. The algorithm adds and deletes array entries efficiently and also keeps the array sorted so as to group similar invalidations into batches.
During an invalidation, a new invalidation function iterates domain->invs, and converts each entry to the corresponding invalidation command(s). This new function is fully compatible with all the existing use cases, allowing a simple rework/replacement.
Some races to keep in mind:
1) A domain can be shared across SMMU instances. When an SMMU instance is removed, the updated invs array has to be sync-ed via synchronize_rcu() to prevent an concurrent invalidation routine that is accessing the old array from issuing commands to the removed SMMU instance.
2) When there are concurrent IOPTE changes (followed by invalidations) and a domain attachment, the new attachment must not become out of sync at the HW level, meaning that an STE store and invalidation array load must be sequenced by the CPU's memory model.
3) When an ATS-enabled device attaches to a blocking domain, the core code requires a hard fence to ensure all ATS invalidations to the device are completed. Relying on RCU alone requires calling synchronize_rcu() that can be too slow. Instead, when ATS is in use, hold a conditional rwlock till all concurrent invalidations are finished.
Related future work and dependent projects:
* NVIDIA is building systems with > 10 SMMU instances where > 8 are being used concurrently in a single VM. So having 8 copies of an identical S2 page table is not efficient. Instead, all vSMMU instances should check compatibility on a shared S2 iopt, to eliminate 7 copies.
Previous attempt based on the list/spinlock design:
iommu/arm-smmu-v3: Allocate vmid per vsmmu instead of s2_parent
https://lore.kernel.org/all/[email protected]/
now can adopt this invs array, avoiding adding complex lists/locks.
* The guest support for BTM requires temporarily invalidating two ASIDs for a single instance. When it renumbers ASIDs this can now be done via the invs array.
* SVA with multiple devices being used by a single process (NVIDIA today has 4-8) sequentially iterates the invalidations through all instances. This ignores the HW concurrency available in each instance. It would be nice to not spin on each sync but go forward and issue batches to other instances also. Reducing to a single SVA domain shared across instances is required to look at this.
This is on Github:
https://github.com/nicolinc/iommufd/commits/arm_smmu_invs-v3
Changelog
v3:
* Add Reviewed/Acked-by from Jason and Balbir
* Rebase on v6.18-rc1
* Drop arm_smmu_invs_dbg()
* Improve kdocs and inline/commit comments
* Rename arm_smmu_invs_cmp to arm_smmu_inv_cmp
* Rename arm_smmu_invs_merge_cmp to arm_smmu_invs_cmp
* Call arm_smmu_invs_flush_iotlb_tags() from arm_smmu_invs_unref()
* Unconditionally trim the invs->num_invs inside arm_smmu_invs_unref(), and simplify arm_smmu_install_old_domain_invs().
v2:
https://lore.kernel.org/all/[email protected]/
* Rebase on v6.17-rc5
* Improve kdocs and inline comments
* Add arm_smmu_invs_dbg() for tracing
* Use users refcount to replace todel flag
* Initialize num_invs in arm_smmu_invs_alloc()
* Add a struct arm_smmu_inv_state to group invs pointers
* Add in struct arm_smmu_invs two flags (has_ats and old)
* Rename master->invs to master->build_invs, and sort the array
* Rework arm_smmu_domain_inv_range() and arm_smmu_invs_end_batch()
* Copy entries by struct arm_smmu_inv in arm_smmu_master_build_invs()
* Add arm_smmu_invs_flush_iotlb_tags() for IOTLB flush by last device
* Rework three invs mutation helpers, and prioritize use the in-place mutation for detach
* Take writer's lock unconditionally but keep it short, and only take reader's lock conditionally on a has_ats flag
v1:
https://lore.kernel.org/all/[email protected]/
Thanks
Nicolin
Jason Gunthorpe (1):
iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
Nicolin Chen (6):
iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free() iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range() iommu/arm-smmu-v3: Perform per-domain invalidations using
arm_smmu_invs
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 135 ++- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 32 +- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 93 ++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 839 +++++++++++++++--- 4 files changed, 924 insertions(+), 175 deletions(-)
Signature · 1 line
2.43.0
PATCH v3 1/7 iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA
Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB.
Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array.
There is no function change.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ 3 files changed, 5 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -0
@@ -858,6 +858,7 @@ struct arm_smmu_master {enum arm_smmu_domain_stage {ARM_SMMU_DOMAIN_S1 =0 ,ARM_SMMU_DOMAIN_S2 , +ARM_SMMU_DOMAIN_SVA , };struct arm_smmu_domain {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -0
@@ -346,6 +346,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,* ARM_SMMU_FEAT_RANGE_INV is present */ smmu_domain->domain.pgsize_bitmap =PAGE_SIZE ; + smmu_domain->stage =ARM_SMMU_DOMAIN_SVA ; smmu_domain->smmu = smmu; ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -0
@@ -3064,6 +3064,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)arm_smmu_install_ste_for_dev (master, &target);arm_smmu_clear_cd (master,IOMMU_NO_PASID );break ; +default : +WARN_ON (true); +break ; }arm_smmu_attach_commit (&state);
Signature · 1 line
2.43.0
PATCH v3 2/7 iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free()
In reply to Nicolin ChenThere will be a bit more things to free than smmu_domain itself. So keep a simple inline function in the header to share aross files.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 ++-- 3 files changed, 8 insertions(+), 3 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +5 -0
@@ -954,6 +954,11 @@ extern struct mutex arm_smmu_asid_lock;struct arm_smmu_domain *arm_smmu_domain_alloc (void ); +static inline void arm_smmu_domain_free (struct arm_smmu_domain *smmu_domain) +{ +kfree (smmu_domain); +} +void arm_smmu_clear_cd (struct arm_smmu_master *master,ioasid_t ssid);struct arm_smmu_cd *arm_smmu_get_cd_ptr (struct arm_smmu_master *master,u32 ssid);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -1
@@ -365,6 +365,6 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, err_asid:xa_erase (&arm_smmu_asid_xa, smmu_domain->cd.asid); err_free: -kfree (smmu_domain); +arm_smmu_domain_free (smmu_domain);return ERR_PTR (ret); }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -2
@@ -2492,7 +2492,7 @@ static void arm_smmu_domain_free_paging(struct iommu_domain *domain)ida_free (&smmu->vmid_map, cfg->vmid); } -kfree (smmu_domain); +arm_smmu_domain_free (smmu_domain); } static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
@@ -3353,7 +3353,7 @@ arm_smmu_domain_alloc_paging_flags(struct device *dev, u32 flags,return &smmu_domain->domain; err_free: -kfree (smmu_domain); +arm_smmu_domain_free (smmu_domain);return ERR_PTR (ret); }
Signature · 1 line
2.43.0
PATCH v3 3/7 iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
In reply to Nicolin ChenFrom: Jason Gunthorpe <[email protected]>
Create a new data structure to hold an array of invalidations that need to be performed for the domain based on what masters are attached, to replace the single smmu pointer and linked list of masters in the current design.
Each array entry holds one of the invalidation actions - S1_ASID, S2_VMID, ATS or their variant with information to feed invalidation commands to HW. It is structured so that multiple SMMUs can participate in the same array, removing one key limitation of the current system.
To maximize performance, a sorted array is used as the data structure. It allows grouping SYNCs together to parallelize invalidations. For instance, it will group all the ATS entries after the ASID/VMID entry, so they will all be pushed to the PCI devices in parallel with one SYNC.
To minimize the locking cost on the invalidation fast path (reader of the invalidation array), the array is managed with RCU.
Provide a set of APIs to add/delete entries to/from an array, which cover cannot-fail attach cases, e.g. attaching to arm_smmu_blocked_domain. Also add kunit coverage for those APIs.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 90 +++++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 93 +++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 242 ++++++++++++++++++ 3 files changed, 425 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +90 -0
@@ -649,6 +649,93 @@ struct arm_smmu_cmdq_batch {int num; }; +/* +* The order here also determines the sequence in which commands are sent to the +* command queue. E.g. TLBI must be done before ATC_INV. +*/ +enum arm_smmu_inv_type { +INV_TYPE_S1_ASID , +INV_TYPE_S2_VMID , +INV_TYPE_S2_VMID_S1_CLEAR , +INV_TYPE_ATS , +INV_TYPE_ATS_FULL , +}; + +struct arm_smmu_inv { +struct arm_smmu_device *smmu; +u8 type; +u8 size_opcode; +u8 nsize_opcode; +u32 id;/* ASID or VMID or SID */ +union { +size_t pgsize;/* ARM_SMMU_FEAT_RANGE_INV */ +u32 ssid;/* INV_TYPE_ATS */ + }; + +refcount_t users;/* users=0 to mark as a trash to be purged */ +}; + +static inline bool arm_smmu_inv_is_ats (struct arm_smmu_inv *inv) +{ +return inv->type ==INV_TYPE_ATS || inv->type ==INV_TYPE_ATS_FULL ; +} + +/** +* struct arm_smmu_invs - Per-domain invalidation array +* @num_invs: number of invalidations in the flexible array +* @rwlock: optional rwlock to fench ATS operations +* @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL +* @rcu: rcu head for kfree_rcu() +* @inv: flexible invalidation array +* +* The arm_smmu_invs is an RCU data structure. During a ->attach_dev callback, +* arm_smmu_invs_merge(), arm_smmu_invs_unref() and arm_smmu_invs_purge() will +* be used to allocate a new copy of an old array for addition and deletion in +* the old domain's and new domain's invs arrays. +* +* The arm_smmu_invs_unref() mutates a given array, by internally reducing the +* users counts of some given entries. This exists to support a no-fail routine +* like attaching to an IOMMU_DOMAIN_BLOCKED. And it could pair with a followup +* arm_smmu_invs_purge() call to generate a new clean array. +* +* Concurrent invalidation thread will push every invalidation described in the +* array into the command queue for each invalidation event. It is designed like +* this to optimize the invalidation fast path by avoiding locks. +* +* A domain can be shared across SMMU instances. When an instance gets removed, +* it would delete all the entries that belong to that SMMU instance. Then, a +* synchronize_rcu() would have to be called to sync the array, to prevent any +* concurrent invalidation thread accessing the old array from issuing commands +* to the command queue of a removed SMMU instance. +*/ +struct arm_smmu_invs { +size_t num_invs; +rwlock_t rwlock; +bool has_ats; +struct rcu_head rcu; +struct arm_smmu_inv inv[]; +}; + +static inline struct arm_smmu_invs *arm_smmu_invs_alloc (size_t num_invs) +{ +struct arm_smmu_invs *new_invs; + + new_invs =kzalloc (struct_size (new_invs, inv, num_invs),GFP_KERNEL ); +if (!new_invs) +return ERR_PTR (-ENOMEM ); +rwlock_init (&new_invs->rwlock); + new_invs->num_invs = num_invs; +return new_invs; +} + +struct arm_smmu_invs *arm_smmu_invs_merge (struct arm_smmu_invs *invs, +struct arm_smmu_invs *to_merge); +size_t arm_smmu_invs_unref (struct arm_smmu_invs *invs, +struct arm_smmu_invs *to_unref, +void (*flush_fn)(struct arm_smmu_inv *inv)); +struct arm_smmu_invs *arm_smmu_invs_purge (struct arm_smmu_invs *invs, +size_t num_trashes); +struct arm_smmu_evtq {struct arm_smmu_queue q;struct iopf_queue *iopf;
@@ -875,6 +962,8 @@ struct arm_smmu_domain {struct iommu_domain domain; +struct arm_smmu_invs __rcu *invs; +/* List of struct arm_smmu_master_domain */ struct list_head devices;spinlock_t devices_lock;
@@ -956,6 +1045,7 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void);static inline void arm_smmu_domain_free (struct arm_smmu_domain *smmu_domain) { +kfree_rcu (smmu_domain->invs, rcu);kfree (smmu_domain); }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c +93 -0
105 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +242 -0
267 lines not on this page · raw message
Signature · 1 line
2.43.0
kernel test robot
build botRe: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
kernel test robot noticed the following build warnings:
Hi Nicolin,
kernel test robot noticed the following build warnings:
[auto build test WARNING on soc/for-next]
[also build test WARNING on linus/master v6.18-rc1 next-20251016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nicolin-Chen/iommu-arm-smmu-v3-Explicitly-set-smmu_domain-stage-for-SVA/20251016-034754 base: https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git for-next patch link: https://lore.kernel.org/r/345bb7703ebd19992694758b47e371900267fa0e.1760555863.git.nicolinc%40nvidia.com patch subject: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array config: arm64-randconfig-001-20251017 (https://download.01.org/0day-ci/archive/20251017/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251017/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1170:7: warning: variable 'cmp' is uninitialized when used here [-Wuninitialized]
1170 | if (cmp <= 0 && !refcount_read(&invs->inv[i].users)) {
| ^~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1167:10: note: initialize the variable 'cmp' to silence this warning
1167 | int cmp;
| ^
| = 01 warning generated.
vim +/cmp +1170 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
1132
1133 /**
1134 * arm_smmu_invs_unref() - Find in @invs for all entries in @to_unref, decrease
1135 * the user counts without deletions
1136 * @invs: the base invalidation array
1137 * @to_unref: an array of invlidations to decrease their user counts
1138 * @flush_fn: A callback function to invoke, when an entry's user count reduces
1139 * to 0
1140 *
1141 * Return: the number of trash entries in the array, for arm_smmu_invs_purge()
1142 *
1143 * This function will not fail. Any entry with users=0 will be marked as trash.
1144 * All tailing trash entries in the array will be dropped. And the size of the
1145 * array will be trimmed properly. All trash entries in-between will remain in
1146 * the @invs until being completely deleted by the next arm_smmu_invs_merge()
1147 * or an arm_smmu_invs_purge() function call.
1148 *
1149 * This function must be locked and serialized with arm_smmu_invs_merge() and
1150 * arm_smmu_invs_purge(), but do not lockdep on any mutex for KUNIT test.
1151 *
1152 * Note that the final @invs->num_invs might not reflect the actual number of
1153 * invalidations due to trash entries. Any reader should take the read lock to
1154 * iterate each entry and check its users counter till the last entry.
1155 */
1156 VISIBLE_IF_KUNIT
1157 size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
1158 struct arm_smmu_invs *to_unref,
1159 void (*flush_fn)(struct arm_smmu_inv *inv))
1160 {
1161 unsigned long flags;
1162 size_t num_trashes = 0;
1163 size_t num_invs = 0;
1164 size_t i, j;
1165
1166 for (i = j = 0; i != invs->num_invs || j != to_unref->num_invs;) {
1167 int cmp;
1168
1169 /* Skip any existing trash entry */
> 1170 if (cmp <= 0 && !refcount_read(&invs->inv[i].users)) { 1171 num_trashes++;
1172 i++;
1173 continue;
1174 }
1175
1176 cmp = arm_smmu_invs_cmp(invs, i, to_unref, j);
1177 if (cmp < 0) {
1178 /* not found in to_unref, leave alone */
1179 i++;
1180 num_invs = i;
1181 } else if (cmp == 0) {
1182 /* same item */
1183 if (refcount_dec_and_test(&invs->inv[i].users)) {
1184 /* KUNIT test doesn't pass in a flush_fn */
1185 if (flush_fn)
1186 flush_fn(&invs->inv[i]);
1187 num_trashes++;
1188 } else {
1189 num_invs = i + 1;
1190 }
1191 i++;
1192 j++;
1193 } else {
1194 /* item in to_unref is not in invs or already a trash */
1195 WARN_ON(true);
1196 j++;
1197 }
1198 }
1199
1200 /* Exclude any tailing trash */
1201 num_trashes -= invs->num_invs - num_invs;
1202
1203 /* The lock is required to fence concurrent ATS operations. */
1204 write_lock_irqsave(&invs->rwlock, flags);
1205 WRITE_ONCE(invs->num_invs, num_invs); /* Remove tailing trash entries */
1206 write_unlock_irqrestore(&invs->rwlock, flags);
1207
1208 return num_trashes;
1209 }
1210 EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_unref);
1211
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
In reply to Nicolin ChenQuoted 10 lines · Nicolin Chen
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10
-0+size_t arm_smmu_invs_unref (struct arm_smmu_invs *invs, +struct arm_smmu_invs *to_unref, +void (*flush_fn)(struct arm_smmu_inv *inv)) +{ +unsigned long flags; +size_t num_trashes =0 ; +size_t num_invs =0 ; +size_t i, j; + +for (i = j =0 ; i != invs->num_invs || j != to_unref->num_invs;) {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_page_response(struct device *dev, struct iopf_fault *unused · line 1176
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10
-0+int cmp; + +/* Skip any existing trash entry */ +if (cmp <=0 && !refcount_read (&invs->inv[i].users)) { + num_trashes++; + i++; +continue ; + } + + cmp =arm_smmu_invs_cmp (invs, i, to_unref, j);
This should be moved upwards to "int cmp" before if.
Will fix in v4.
Nicolin
kernel test robot
build botRe: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
In reply to Nicolin Chenkernel test robot noticed the following build warnings:
Hi Nicolin,
kernel test robot noticed the following build warnings:
[auto build test WARNING on soc/for-next]
[also build test WARNING on linus/master v6.18-rc1 next-20251016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nicolin-Chen/iommu-arm-smmu-v3-Explicitly-set-smmu_domain-stage-for-SVA/20251016-034754 base: https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git for-next patch link: https://lore.kernel.org/r/345bb7703ebd19992694758b47e371900267fa0e.1760555863.git.nicolinc%40nvidia.com patch subject: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array config: arm64-randconfig-r123-20251017 (https://download.01.org/0day-ci/archive/20251017/[email protected]/config) compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 754ebc6ebb9fb9fbee7aef33478c74ea74949853) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251017/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c: note: in included file:
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct callback_head *head @@ got struct callback_head [noderef] __rcu * @@ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: expected struct callback_head *head drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: got struct callback_head [noderef] __rcu * >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: cast removes address space '__rcu' of expression
--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c: note: in included file (through arch/arm64/include/asm/atomic.h, include/linux/atomic.h, include/asm-generic/bitops/atomic.h, ...):
arch/arm64/include/asm/cmpxchg.h:168:1: sparse: sparse: cast truncates bits from constant value (ffffffff80000000 becomes 0) arch/arm64/include/asm/cmpxchg.h:168:1: sparse: sparse: cast truncates bits from constant value (ffffffff80000000 becomes 0) drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c: note: in included file:
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct callback_head *head @@ got struct callback_head [noderef] __rcu * @@ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: expected struct callback_head *head drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: got struct callback_head [noderef] __rcu * >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: cast removes address space '__rcu' of expression >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct callback_head *head @@ got struct callback_head [noderef] __rcu * @@ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: expected struct callback_head *head drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: got struct callback_head [noderef] __rcu * >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: cast removes address space '__rcu' of expression
vim +1048 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
1045
1046 static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domain)
1047 {
> 1048 kfree_rcu(smmu_domain->invs, rcu);
1049 kfree(smmu_domain);
1050 }
1051
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
Quoted 5 lines · kernel test robot
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c: note: in included file:Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected structcallback_head*head @@ got structcallback_head[noderef]__rcu* @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: expected structcallback_head*head
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: got structcallback_head[noderef]__rcu*Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: cast removes address space '__rcu' of expression
...
Quoted 4 lines
1045
1046 static inline voidarm_smmu_domain_free(struct arm_smmu_domain *smmu_domain)
1047 {Quoted 1 line
1048
kfree_rcu(smmu_domain->invs, rcu);
Looks like it should be:
static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domain) {
- kfree_rcu(smmu_domain->invs, rcu);
+ struct arm_smmu_invs *invs = rcu_dereference(smmu_domain->invs);
+
+ kfree_rcu(invs, rcu);
kfree(smmu_domain);}
Will fix.
Thanks
Nicolin
Re: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
Quoted 15 lines · Nicolin Chen
Quoted 5 lines · kernel test robot
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c: note: in included file:Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected structcallback_head*head @@ got structcallback_head[noderef]__rcu* @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: expected structcallback_head*head
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: got structcallback_head[noderef]__rcu*Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: cast removes address space '__rcu' of expression...
Quoted 4 lines
1045
1046 static inline voidarm_smmu_domain_free(struct arm_smmu_domain *smmu_domain)
1047 {Quoted 1 line
1048
kfree_rcu(smmu_domain->invs, rcu);Looks like it should be:
static inline voidarm_smmu_domain_free(struct arm_smmu_domain *smmu_domain){
Review
Quoted 2 lines
(quoted diff) +1
-1- kfree_rcu(smmu_domain->invs, rcu); + struct arm_smmu_invs *invs = rcu_dereference(smmu_domain->invs);
rcu_derference_protected(,true) since we know there is no concurrency here..
Jason
Re: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Nicolin Chen
Quoted 5 lines · kernel test robot
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c: note: in included file:Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected structcallback_head*head @@ got structcallback_head[noderef]__rcu* @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: expected structcallback_head*head
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: got structcallback_head[noderef]__rcu*Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1048:9: sparse: sparse: cast removes address space '__rcu' of expression...
Quoted 4 lines
1045
1046 static inline voidarm_smmu_domain_free(struct arm_smmu_domain *smmu_domain)
1047 {Quoted 1 line
1048
kfree_rcu(smmu_domain->invs, rcu);
Quoted 8 lines
Review
Quoted 5 lines
Looks like it should be:
static inline voidarm_smmu_domain_free(struct arm_smmu_domain *smmu_domain){(quoted diff) +1
-1- kfree_rcu(smmu_domain->invs, rcu); + struct arm_smmu_invs *invs = rcu_dereference(smmu_domain->invs);
rcu_derference_protected(,true)since we know there is no concurrency here..
Oh right, it's outside rcu_read_lock().
Thanks
Nicolin
kernel test robot
build botRe: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
In reply to Nicolin Chenkernel test robot noticed the following build errors:
Hi Nicolin,
kernel test robot noticed the following build errors:
[auto build test ERROR on soc/for-next]
[also build test ERROR on linus/master v6.18-rc3 next-20251027] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nicolin-Chen/iommu-arm-smmu-v3-Explicitly-set-smmu_domain-stage-for-SVA/20251016-034754 base: https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git for-next patch link: https://lore.kernel.org/r/345bb7703ebd19992694758b47e371900267fa0e.1760555863.git.nicolinc%40nvidia.com patch subject: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array config: arm64-defconfig (https://download.01.org/0day-ci/archive/20251027/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 15.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251027/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1062:23: error: static declaration of 'arm_smmu_invs_merge' follows non-static declaration
1062 | struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
In file included from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:34:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:731:23: note: previous declaration of 'arm_smmu_invs_merge' with type 'struct arm_smmu_invs *(struct arm_smmu_invs *, struct arm_smmu_invs *)'
731 | struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1157:8: error: static declaration of 'arm_smmu_invs_unref' follows non-static declaration
1157 | size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:733:8: note: previous declaration of 'arm_smmu_invs_unref' with type 'size_t(struct arm_smmu_invs *, struct arm_smmu_invs *, void (*)(struct arm_smmu_inv *))' {aka 'long unsigned int(struct arm_smmu_invs *, struct arm_smmu_invs *, void (*)(struct arm_smmu_inv *))'}
733 | size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1227:23: error: static declaration of 'arm_smmu_invs_purge' follows non-static declaration 1227 | struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:736:23: note: previous declaration of 'arm_smmu_invs_purge' with type 'struct arm_smmu_invs *(struct arm_smmu_invs *, size_t)' {aka 'struct arm_smmu_invs *(struct arm_smmu_invs *, long unsigned int)'}
736 | struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1227:23: warning: 'arm_smmu_invs_purge' defined but not used [-Wunused-function]
1227 | struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1157:8: warning: 'arm_smmu_invs_unref' defined but not used [-Wunused-function]
1157 | size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1062:23: warning: 'arm_smmu_invs_merge' defined but not used [-Wunused-function]
1062 | struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
| ^~~~~~~~~~~~~~~~~~~
vim +/arm_smmu_invs_merge +1062 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
1043
1044 /**
1045 * arm_smmu_invs_merge() - Merge @to_merge into @invs and generate a new array
1046 * @invs: the base invalidation array
1047 * @to_merge: an array of invlidations to merge
1048 *
1049 * Return: a newly allocated array on success, or ERR_PTR
1050 *
1051 * This function must be locked and serialized with arm_smmu_invs_unref() and
1052 * arm_smmu_invs_purge(), but do not lockdep on any lock for KUNIT test.
1053 *
1054 * Both @invs and @to_merge must be sorted, to ensure the returned array will be
1055 * sorted as well.
1056 *
1057 * Caller is resposible for freeing the @invs and the returned new one.
1058 *
1059 * Entries marked as trash will be purged in the returned array.
1060 */
1061 VISIBLE_IF_KUNIT
> 1062 struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
1063 struct arm_smmu_invs *to_merge)
1064 {
1065 struct arm_smmu_invs *new_invs;
1066 struct arm_smmu_inv *new;
1067 size_t num_trashes = 0;
1068 size_t num_adds = 0;
1069 size_t i, j;
1070
1071 for (i = j = 0; i != invs->num_invs || j != to_merge->num_invs;) {
1072 int cmp = arm_smmu_invs_cmp(invs, i, to_merge, j);
1073
1074 /* Skip any unwanted trash entry */
1075 if (cmp < 0 && !refcount_read(&invs->inv[i].users)) {
1076 num_trashes++;
1077 i++;
1078 continue;
1079 }
1080
1081 if (cmp < 0) {
1082 /* not found in to_merge, leave alone */
1083 i++;
1084 } else if (cmp == 0) {
1085 /* same item */
1086 i++;
1087 j++;
1088 } else {
1089 /* unique to to_merge */
1090 num_adds++;
1091 j++;
1092 }
1093 }
1094
1095 new_invs = arm_smmu_invs_alloc(invs->num_invs - num_trashes + num_adds);
1096 if (IS_ERR(new_invs))
1097 return new_invs;
1098
1099 new = new_invs->inv;
1100 for (i = j = 0; i != invs->num_invs || j != to_merge->num_invs;) {
1101 int cmp = arm_smmu_invs_cmp(invs, i, to_merge, j);
1102
1103 if (cmp <= 0 && !refcount_read(&invs->inv[i].users)) {
1104 i++;
1105 continue;
1106 }
1107
1108 if (cmp < 0) {
1109 *new = invs->inv[i];
1110 i++;
1111 } else if (cmp == 0) {
1112 *new = invs->inv[i];
1113 refcount_inc(&new->users);
1114 i++;
1115 j++;
1116 } else {
1117 *new = to_merge->inv[j];
1118 refcount_set(&new->users, 1);
1119 j++;
1120 }
1121
1122 if (new != new_invs->inv)
1123 WARN_ON_ONCE(arm_smmu_inv_cmp(new - 1, new) == 1);
1124 new++;
1125 }
1126
1127 WARN_ON(new != new_invs->inv + new_invs->num_invs);
1128
1129 return new_invs;
1130 }
1131 EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge);
1132
1133 /**
1134 * arm_smmu_invs_unref() - Find in @invs for all entries in @to_unref, decrease
1135 * the user counts without deletions
1136 * @invs: the base invalidation array
1137 * @to_unref: an array of invlidations to decrease their user counts
1138 * @flush_fn: A callback function to invoke, when an entry's user count reduces
1139 * to 0
1140 *
1141 * Return: the number of trash entries in the array, for arm_smmu_invs_purge()
1142 *
1143 * This function will not fail. Any entry with users=0 will be marked as trash.
1144 * All tailing trash entries in the array will be dropped. And the size of the
1145 * array will be trimmed properly. All trash entries in-between will remain in
1146 * the @invs until being completely deleted by the next arm_smmu_invs_merge()
1147 * or an arm_smmu_invs_purge() function call.
1148 *
1149 * This function must be locked and serialized with arm_smmu_invs_merge() and
1150 * arm_smmu_invs_purge(), but do not lockdep on any mutex for KUNIT test.
1151 *
1152 * Note that the final @invs->num_invs might not reflect the actual number of
1153 * invalidations due to trash entries. Any reader should take the read lock to
1154 * iterate each entry and check its users counter till the last entry.
1155 */
1156 VISIBLE_IF_KUNIT
> 1157 size_t arm_smmu_invs_unref(struct arm_smmu_invs *invs,
1158 struct arm_smmu_invs *to_unref,
1159 void (*flush_fn)(struct arm_smmu_inv *inv))
1160 {
1161 unsigned long flags;
1162 size_t num_trashes = 0;
1163 size_t num_invs = 0;
1164 size_t i, j;
1165
1166 for (i = j = 0; i != invs->num_invs || j != to_unref->num_invs;) {
1167 int cmp;
1168
1169 /* Skip any existing trash entry */
1170 if (cmp <= 0 && !refcount_read(&invs->inv[i].users)) {
1171 num_trashes++;
1172 i++;
1173 continue;
1174 }
1175
1176 cmp = arm_smmu_invs_cmp(invs, i, to_unref, j);
1177 if (cmp < 0) {
1178 /* not found in to_unref, leave alone */
1179 i++;
1180 num_invs = i;
1181 } else if (cmp == 0) {
1182 /* same item */
1183 if (refcount_dec_and_test(&invs->inv[i].users)) {
1184 /* KUNIT test doesn't pass in a flush_fn */
1185 if (flush_fn)
1186 flush_fn(&invs->inv[i]);
1187 num_trashes++;
1188 } else {
1189 num_invs = i + 1;
1190 }
1191 i++;
1192 j++;
1193 } else {
1194 /* item in to_unref is not in invs or already a trash */
1195 WARN_ON(true);
1196 j++;
1197 }
1198 }
1199
1200 /* Exclude any tailing trash */
1201 num_trashes -= invs->num_invs - num_invs;
1202
1203 /* The lock is required to fence concurrent ATS operations. */
1204 write_lock_irqsave(&invs->rwlock, flags);
1205 WRITE_ONCE(invs->num_invs, num_invs); /* Remove tailing trash entries */
1206 write_unlock_irqrestore(&invs->rwlock, flags);
1207
1208 return num_trashes;
1209 }
1210 EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_unref);
1211
1212 /**
1213 * arm_smmu_invs_purge() - Purge all the trash entries in the @invs
1214 * @invs: the base invalidation array
1215 * @num_trashes: expected number of trash entries, typically returned by a prior
1216 * arm_smmu_invs_unref() call
1217 *
1218 * Return: a newly allocated array on success removing all the trash entries, or
1219 * NULL on failure
1220 *
1221 * This function must be locked and serialized with arm_smmu_invs_merge() and
1222 * arm_smmu_invs_unref(), but do not lockdep on any lock for KUNIT test.
1223 *
1224 * Caller is resposible for freeing the @invs and the returned new one.
1225 */
1226 VISIBLE_IF_KUNIT
> 1227 struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs, 1228 size_t num_trashes)
1229 {
1230 struct arm_smmu_invs *new_invs;
1231 size_t i, j;
1232
1233 if (WARN_ON(invs->num_invs < num_trashes))
1234 return NULL;
1235
1236 new_invs = arm_smmu_invs_alloc(invs->num_invs - num_trashes);
1237 if (IS_ERR(new_invs))
1238 return NULL;
1239
1240 for (i = j = 0; i != invs->num_invs; i++) {
1241 if (!refcount_read(&invs->inv[i].users))
1242 continue;
1243 new_invs->inv[j] = invs->inv[i];
1244 j++;
1245 }
1246
1247 WARN_ON(j != new_invs->num_invs);
1248 return new_invs;
1249 }
1250 EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_purge);
1251
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array
Quoted 7 lines · kernel test robot
Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1062:23: error: static declaration of 'arm_smmu_invs_merge' follows non-static declaration1062 | struct
arm_smmu_invs*arm_smmu_invs_merge(structarm_smmu_invs*invs,
| ^~~~~~~~~~~~~~~~~~~
In file included fromdrivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:34:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:731:23: note: previous declaration of 'arm_smmu_invs_merge' with type 'structarm_smmu_invs*(structarm_smmu_invs*, structarm_smmu_invs*)'
731 | structarm_smmu_invs*arm_smmu_invs_merge(structarm_smmu_invs*invs,
| ^~~~~~~~~~~~~~~~~~~
These should be added under "#if IS_ENABLED(CONFIG_KUNIT)" in the header.
I will fix this and send a v4 today.
Nicolin
PATCH v3 4/7 iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array
In reply to Nicolin ChenWhen a master is attached from an old domain to a new domain, it needs to build an invalidation array to delete and add the array entries from/onto the invalidation arrays of those two domains, passed via the to_merge and to_unref arguments into arm_smmu_invs_merge/unref() respectively.
Since the master->num_streams might differ across masters, a memory would have to be allocated when building an to_merge/to_unref array which might fail with -ENOMEM.
On the other hand, an attachment to arm_smmu_blocked_domain must not fail so it's the best to avoid any memory allocation in that path.
Pre-allocate a fixed size invalidation array for every master. This array will be used as a scratch to fill dynamically when building a to_merge or to_unref invs array. Sort fwspec->ids in an ascending order to fit to the arm_smmu_invs_merge() function.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 ++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 27 +++++++++++++++++++++ 2 files changed, 35 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8 -0
@@ -930,6 +930,14 @@ struct arm_smmu_master {struct arm_smmu_device *smmu;struct device *dev;struct arm_smmu_stream *streams; +/* +* Scratch memory for a to_merge or to_unref array to build a per-domain +* invalidation array. It'll be pre-allocated with enough enries for all +* possible build scenarios. It can be used by only one caller at a time +* until the arm_smmu_invs_merge/unref() finishes. Must be locked by the +* iommu_group mutex. +*/ +struct arm_smmu_invs *build_invs;struct arm_smmu_vmaster *vmaster;/* use smmu->streams_mutex */ /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +27 -0
@@ -3687,12 +3687,22 @@ static int arm_smmu_init_sid_strtab(struct arm_smmu_device *smmu, u32 sid)return 0 ; } +static int arm_smmu_ids_cmp (const void *_l,const void *_r) +{ + const typeof_member(struct iommu_fwspec, ids[0]) *l = _l; + const typeof_member(struct iommu_fwspec, ids[0]) *r = _r; + +return cmp_int (*l, *r); +} +static int arm_smmu_insert_master (struct arm_smmu_device *smmu,struct arm_smmu_master *master) {int i;int ret =0 ;struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (master->dev); +bool ats_supported =dev_is_pci (master->dev) && +pci_ats_supported (to_pci_dev (master->dev)); master->streams =kcalloc (fwspec->num_ids,sizeof (*master->streams),GFP_KERNEL );
@@ -3700,6 +3710,21 @@ static int arm_smmu_insert_master(struct arm_smmu_device *smmu,return -ENOMEM ; master->num_streams = fwspec->num_ids; +if (!ats_supported) { +/* Base case has 1 ASID entry or maximum 2 VMID entries */ + master->build_invs =arm_smmu_invs_alloc (2 ); + }else { +/* Put the ids into order for sorted to_merge/to_unref arrays */ +sort_nonatomic (fwspec->ids, fwspec->num_ids, +sizeof (fwspec->ids[0 ]), arm_smmu_ids_cmp,NULL ); +/* ATS case adds num_ids of entries, on top of the base case */ + master->build_invs =arm_smmu_invs_alloc (2 + fwspec->num_ids); + } +if (IS_ERR (master->build_invs)) { +kfree (master->streams); +return PTR_ERR (master->build_invs); + } +mutex_lock (&smmu->streams_mutex);for (i =0 ; i < fwspec->num_ids; i++) {struct arm_smmu_stream *new_stream = &master->streams[i];
@@ -3737,6 +3762,7 @@ static int arm_smmu_insert_master(struct arm_smmu_device *smmu,for (i--; i >=0 ; i--)rb_erase (&master->streams[i].node, &smmu->streams);kfree (master->streams); +kfree (master->build_invs); }mutex_unlock (&smmu->streams_mutex);
@@ -3758,6 +3784,7 @@ static void arm_smmu_remove_master(struct arm_smmu_master *master)mutex_unlock (&smmu->streams_mutex);kfree (master->streams); +kfree (master->build_invs); }static struct iommu_device *arm_smmu_probe_device (struct device *dev)
Signature · 1 line
2.43.0
PATCH v3 5/7 iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters
In reply to Nicolin ChenUpdate the invs array with the invalidations required by each domain type during attachment operations.
Only an SVA domain or a paging domain will have an invs array:
a. SVA domain will add an INV_TYPE_S1_ASID per SMMU and an INV_TYPE_ATS
per SID
b. Non-nesting-parent paging domain with no ATS-enabled master will add
a single INV_TYPE_S1_ASID or INV_TYPE_S2_VMID per SMMU
c. Non-nesting-parent paging domain with ATS-enabled master(s) will do
(b) and add an INV_TYPE_ATS per SID
d. Nesting-parent paging domain will add an INV_TYPE_S2_VMID followed by
an INV_TYPE_S2_VMID_S1_CLEAR per vSMMU. For an ATS-enabled master, it will add an INV_TYPE_ATS_FULL per SID
Note that case #d prepares for a future implementation of VMID allocation which requires a followup series for S2 domain sharing. So when a nesting parent domain is attached through a vSMMU instance using a nested domain. VMID will be allocated per vSMMU instance v.s. currectly per S2 domain.
The per-domain invalidation is not needed until the domain is attached to a master (when it starts to possibly use TLB). This will make it possible to attach the domain to multiple SMMUs and avoid unnecessary invalidation overhead during teardown if no STEs/CDs refer to the domain. It also means that when the last device is detached, the old domain must flush its ASID or VMID, since any new iommu_unmap() call would not trigger invalidations given an empty domain->invs array.
Introduce some arm_smmu_invs helper functions for building scratch arrays, preparing and installing old/new domain's invalidation arrays.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 17 ++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 242 +++++++++++++++++++- 2 files changed, 258 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +17 -0
@@ -1093,6 +1093,21 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)IOMMU_FWSPEC_PCI_RC_CANWBS ; } +/** +* struct arm_smmu_inv_state - Per-domain invalidation array state +* @invs_ptr: points to the domain->invs (unwinding nesting/etc.) or is NULL if +* no change should be made +* @old_invs: the original invs array +* @new_invs: for new domain, this is the new invs array to update domin->invs; +* for old domain, this is the master->build_invs to pass in as the +* to_unref argument to an arm_smmu_invs_unref() call +*/ +struct arm_smmu_inv_state { +struct arm_smmu_invs **invs_ptr; +struct arm_smmu_invs *old_invs; +struct arm_smmu_invs *new_invs; +}; +struct arm_smmu_attach_state {/* Inputs */ struct iommu_domain *old_domain;
@@ -1102,6 +1117,8 @@ struct arm_smmu_attach_state {ioasid_t ssid;/* Resulting state */ struct arm_smmu_vmaster *vmaster; +struct arm_smmu_inv_state old_domain_invst; +struct arm_smmu_inv_state new_domain_invst;bool ats_enabled; };
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +241 -1
283 lines not on this page · raw message
Signature · 1 line
2.43.0
kernel test robot
build botRe: [PATCH v3 5/7] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters
kernel test robot noticed the following build warnings:
Hi Nicolin,
kernel test robot noticed the following build warnings:
[auto build test WARNING on soc/for-next]
[also build test WARNING on linus/master v6.18-rc1 next-20251016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nicolin-Chen/iommu-arm-smmu-v3-Explicitly-set-smmu_domain-stage-for-SVA/20251016-034754 base: https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git for-next patch link: https://lore.kernel.org/r/14d76eebae359825442a96c0ffa13687de792063.1760555863.git.nicolinc%40nvidia.com patch subject: [PATCH v3 5/7] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters config: arm64-randconfig-r123-20251017 (https://download.01.org/0day-ci/archive/20251017/[email protected]/config) compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 754ebc6ebb9fb9fbee7aef33478c74ea74949853) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251017/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct arm_smmu_invs **invs_ptr @@ got struct arm_smmu_invs [noderef] __rcu ** @@ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: expected struct arm_smmu_invs **invs_ptr drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: got struct arm_smmu_invs [noderef] __rcu ** drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3226:33: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct arm_smmu_invs **invs_ptr @@ got struct arm_smmu_invs [noderef] __rcu ** @@ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3226:33: sparse: expected struct arm_smmu_invs **invs_ptr drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3226:33: sparse: got struct arm_smmu_invs [noderef] __rcu ** >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3247:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3247:9: sparse: struct arm_smmu_invs [noderef] __rcu *
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3247:9: sparse: struct arm_smmu_invs *
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3305:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3305:9: sparse: struct arm_smmu_invs [noderef] __rcu *
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3305:9: sparse: struct arm_smmu_invs *
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c: note: in included file (through arch/arm64/include/asm/atomic.h, include/linux/atomic.h, include/asm-generic/bitops/atomic.h, ...):
arch/arm64/include/asm/cmpxchg.h:168:1: sparse: sparse: cast truncates bits from constant value (ffffffff80000000 becomes 0) arch/arm64/include/asm/cmpxchg.h:168:1: sparse: sparse: cast truncates bits from constant value (ffffffff80000000 becomes 0) drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c: note: in included file:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct callback_head *head @@ got struct callback_head [noderef] __rcu * @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: expected struct callback_head *head
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: got struct callback_head [noderef] __rcu *
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: sparse: cast removes address space '__rcu' of expression
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct callback_head *head @@ got struct callback_head [noderef] __rcu * @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: expected struct callback_head *head
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: got struct callback_head [noderef] __rcu *
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h:1056:9: sparse: sparse: cast removes address space '__rcu' of expression
vim +3208 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
3174
3175 /*
3176 * During attachment, the updates of the two domain->invs arrays are sequenced:
3177 * 1. new domain updates its invs array, merging master->build_invs
3178 * 2. new domain starts to include the master during its invalidation
3179 * 3. master updates its STE switching from the old domain to the new domain
3180 * 4. old domain still includes the master during its invalidation
3181 * 5. old domain updates its invs array, unreferencing master->build_invs
3182 *
3183 * For 1 and 5, prepare the two updated arrays in advance, handling any changes
3184 * that can possibly failure. So the actual update of either 1 or 5 won't fail.
3185 * arm_smmu_asid_lock ensures that the old invs in the domains are intact while
3186 * we are sequencing to update them.
3187 */
3188 static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *state,
3189 struct arm_smmu_domain *new_smmu_domain)
3190 {
3191 struct arm_smmu_domain *old_smmu_domain =
3192 to_smmu_domain_devices(state->old_domain);
3193 struct arm_smmu_master *master = state->master;
3194 ioasid_t ssid = state->ssid;
3195
3196 /* A re-attach case doesn't need to update invs array */
3197 if (new_smmu_domain == old_smmu_domain)
3198 return 0;
3199
3200 /*
3201 * At this point a NULL domain indicates the domain doesn't use the
3202 * IOTLB, see to_smmu_domain_devices().
3203 */
3204 if (new_smmu_domain) {
3205 struct arm_smmu_inv_state *invst = &state->new_domain_invst;
3206 struct arm_smmu_invs *build_invs;
3207
> 3208 invst->invs_ptr = &new_smmu_domain->invs;
3209 invst->old_invs = rcu_dereference_protected(
3210 new_smmu_domain->invs,
3211 lockdep_is_held(&arm_smmu_asid_lock));
3212 build_invs = arm_smmu_master_build_invs(
3213 master, state->ats_enabled, ssid, new_smmu_domain);
3214 if (!build_invs)
3215 return -EINVAL;
3216
3217 invst->new_invs =
3218 arm_smmu_invs_merge(invst->old_invs, build_invs);
3219 if (IS_ERR(invst->new_invs))
3220 return PTR_ERR(invst->new_invs);
3221 }
3222
3223 if (old_smmu_domain) {
3224 struct arm_smmu_inv_state *invst = &state->old_domain_invst;
3225
3226 invst->invs_ptr = &old_smmu_domain->invs;
3227 invst->old_invs = rcu_dereference_protected(
3228 old_smmu_domain->invs,
3229 lockdep_is_held(&arm_smmu_asid_lock));
3230 /* For old_smmu_domain, new_invs points to master->build_invs */
3231 invst->new_invs = arm_smmu_master_build_invs(
3232 master, master->ats_enabled, ssid, old_smmu_domain);
3233 }
3234
3235 return 0;
3236 }
3237
3238 /* Must be installed before arm_smmu_install_ste_for_dev() */
3239 static void
3240 arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state)
3241 {
3242 struct arm_smmu_inv_state *invst = &state->new_domain_invst;
3243
3244 if (!invst->invs_ptr)
3245 return;
3246
> 3247 rcu_assign_pointer(*invst->invs_ptr, invst->new_invs); 3248 /*
3249 * We are committed to updating the STE. Ensure the invalidation array
3250 * is visable to concurrent map/unmap threads, and acquire any racying
3251 * IOPTE updates.
3252 */
3253 smp_mb();
3254 kfree_rcu(invst->old_invs, rcu);
3255 }
3256
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 5/7] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters
Quoted 4 lines · kernel test robot
sparse warnings: (new ones prefixed by >>)
Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected structarm_smmu_invs**invs_ptr @@ got structarm_smmu_invs[noderef]__rcu** @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: expected structarm_smmu_invs**invs_ptr
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: got structarm_smmu_invs[noderef]__rcu**
...
Quoted 1 line
Quoted 1 line
3208
invst->invs_ptr=&new_smmu_domain->invs;
...
Quoted 1 line
Quoted 1 line
3247
rcu_assign_pointer(*invst->invs_ptr, invst->new_invs);
Looks like we need:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -1
@@ -1105,7 +1105,7 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)* to_unref argument to an arm_smmu_invs_unref() call */ struct arm_smmu_inv_state { -struct arm_smmu_invs **invs_ptr; +struct arm_smmu_invs __rcu **invs_ptr;struct arm_smmu_invs *old_invs;struct arm_smmu_invs *new_invs; };
Will fix in v4.
Thanks
Nicolin
Re: [PATCH v3 5/7] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters
Quoted 17 lines · Nicolin Chen
Quoted 4 lines · kernel test robot
sparse warnings: (new ones prefixed by >>)
Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected structarm_smmu_invs**invs_ptr @@ got structarm_smmu_invs[noderef]__rcu** @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: expected structarm_smmu_invs**invs_ptr
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: got structarm_smmu_invs[noderef]__rcu**...
Quoted 1 line
Quoted 1 line
3208
invst->invs_ptr=&new_smmu_domain->invs;...
Quoted 1 line
Quoted 1 line
3247
rcu_assign_pointer(*invst->invs_ptr, invst->new_invs);Looks like we need:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0
-0
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master) · line 1111
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1
-1@@ -1105,7 +1105,7 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)* to_unref argument to an arm_smmu_invs_unref() call */ struct arm_smmu_inv_state { -struct arm_smmu_invs **invs_ptr; +struct arm_smmu_invs __rcu **invs_ptr;struct arm_smmu_invs *old_invs;struct arm_smmu_invs *new_invs; };
Isn't it:
struct arm_smmu_invs * __rcu *invs_ptr;
This is a pointer to a rcu controlled pointer, not a rcu controlled pointer to a pointer..
Jason
Re: [PATCH v3 5/7] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters
Quoted 29 lines · Jason Gunthorpe
Quoted 17 lines · Nicolin Chen
Quoted 4 lines · kernel test robot
sparse warnings: (new ones prefixed by >>)
Quoted 1 line
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected structarm_smmu_invs**invs_ptr @@ got structarm_smmu_invs[noderef]__rcu** @@
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: expected structarm_smmu_invs**invs_ptr
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3208:33: sparse: got structarm_smmu_invs[noderef]__rcu**...
Quoted 1 line
Quoted 1 line
3208
invst->invs_ptr=&new_smmu_domain->invs;...
Quoted 1 line
Quoted 1 line
3247
rcu_assign_pointer(*invst->invs_ptr, invst->new_invs);Looks like we need:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0
-0drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master) · line 1111
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1
-1@@ -1105,7 +1105,7 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)* to_unref argument to an arm_smmu_invs_unref() call */ struct arm_smmu_inv_state { -struct arm_smmu_invs **invs_ptr; +struct arm_smmu_invs __rcu **invs_ptr;struct arm_smmu_invs *old_invs;struct arm_smmu_invs *new_invs; };Isn't it:
Quoted 1 line
struct
arm_smmu_invs*__rcu*invs_ptr;
Hmm, sparse warns this:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3257:33: warning: incorrect type in assignment (different address spaces)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3257:33: expected struct arm_smmu_invs *[noderef] __rcu *invs_ptrdrivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3257:33: got struct arm_smmu_invs [noderef] __rcu **drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3275:33: warning: incorrect type in assignment (different address spaces)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3275:33: expected struct arm_smmu_invs *[noderef] __rcu *invs_ptrdrivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3275:33: got struct arm_smmu_invs [noderef] __rcu **drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3296:9: error: incompatible types in comparison expression (different address spaces):drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3296:9: struct arm_smmu_invs [noderef] __rcu *drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3296:9: struct arm_smmu_invs *drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3354:9: error: incompatible types in comparison expression (different address spaces):drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3354:9: struct arm_smmu_invs [noderef] __rcu *drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:3354:9: struct arm_smmu_invs *
3257: invst->invs_ptr = &new_smmu_domain->invs; 3296: rcu_assign_pointer(*invst->invs_ptr, invst->new_invs); 3354: rcu_assign_pointer(*invst->invs_ptr, new_invs);
But no warning with "struct arm_smmu_invs __rcu **invs_ptr".
Thanks
Nicolin
PATCH v3 6/7 iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range()
In reply to Nicolin ChenEach smmu_domain now has an arm_smmu_invs that specifies the invalidation steps to perform after any change the IOPTEs. This includes supports for basic ASID/VMID, the special case for nesting, and ATC invalidations.
Introduce a new arm_smmu_domain_inv helper iterating smmu_domain->invs to convert the invalidation array to commands. Any invalidation request with no size specified means an entire flush over a range based one.
Take advantage of the sorted array to compatible batch operations together to the same SMMU. For instance, ATC invaliations for multiple SIDs can be pushed as a batch.
ATC invalidations must be completed before the driver disables ATS. Or the device is permitted to ignore any racing invalidation that would cause an SMMU timeout. The sequencing is done with a rwlock where holding the write side of the rwlock means that there are no outstanding ATC invalidations. If ATS is not used the rwlock is ignored, similar to the existing code.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 218 ++++++++++++++++++-- 2 files changed, 214 insertions(+), 13 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +9 -0
@@ -1078,6 +1078,15 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,int arm_smmu_atc_inv_domain (struct arm_smmu_domain *smmu_domain,unsigned long iova,size_t size); +void arm_smmu_domain_inv_range (struct arm_smmu_domain *smmu_domain, +unsigned long iova,size_t size, +unsigned int granule,bool leaf); + +static inline void arm_smmu_domain_inv (struct arm_smmu_domain *smmu_domain) +{ +arm_smmu_domain_inv_range (smmu_domain,0 ,0 ,0 , false); +} +void __arm_smmu_cmdq_skip_err (struct arm_smmu_device *smmu,struct arm_smmu_cmdq *cmdq); int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +205 -13
251 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v3 7/7 iommu/arm-smmu-v3: Perform per-domain invalidations using arm_smmu_invs
In reply to Nicolin ChenReplace the old invalidation functions with arm_smmu_domain_inv_range() in all the existing invalidation routines. And deprecate the old functions.
The new arm_smmu_domain_inv_range() handles the CMDQ_MAX_TLBI_OPS as well, so drop it in the SVA function.
Since arm_smmu_cmdq_batch_add_range() has only one caller now, and it must be given a valid size, add a WARN_ON_ONCE to catch any missed case.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 - .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 29 +-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 165 +----------------- 3 files changed, 11 insertions(+), 190 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -7
@@ -1071,13 +1071,6 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, struct arm_smmu_cd *cd, struct iommu_domain *old); -void arm_smmu_tlb_inv_asid (struct arm_smmu_device *smmu,u16 asid); -void arm_smmu_tlb_inv_range_asid (unsigned long iova,size_t size,int asid, -size_t granule,bool leaf, -struct arm_smmu_domain *smmu_domain); -int arm_smmu_atc_inv_domain (struct arm_smmu_domain *smmu_domain, -unsigned long iova,size_t size); -void arm_smmu_domain_inv_range (struct arm_smmu_domain *smmu_domain,unsigned long iova,size_t size,unsigned int granule,bool leaf);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +3 -26
@@ -122,15 +122,6 @@ void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, }EXPORT_SYMBOL_IF_KUNIT (arm_smmu_make_sva_cd); -/* -* Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this -* is used as a threshold to replace per-page TLBI commands to issue in the -* command queue with an address-space TLBI command, when SMMU w/o a range -* invalidation feature handles too many per-page TLBI commands, which will -* otherwise result in a soft lockup. -*/ -#define CMDQ_MAX_TLBI_OPS (1 << (PAGE_SHIFT - 3)) - static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start,
@@ -146,21 +137,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,* range. So do a simple translation here by calculating size correctly. */ size = end - start; -if (!(smmu_domain->smmu->features &ARM_SMMU_FEAT_RANGE_INV )) { -if (size >=CMDQ_MAX_TLBI_OPS *PAGE_SIZE ) - size =0 ; - }else { -if (size ==ULONG_MAX ) - size =0 ; - } - -if (!size) -arm_smmu_tlb_inv_asid (smmu_domain->smmu, smmu_domain->cd.asid); -else -arm_smmu_tlb_inv_range_asid (start, size, smmu_domain->cd.asid, -PAGE_SIZE , false, smmu_domain); -arm_smmu_atc_inv_domain (smmu_domain, start, size); +arm_smmu_domain_inv_range (smmu_domain, start, size,PAGE_SIZE , false); }static void arm_smmu_mm_release (struct mmu_notifier *mn,struct mm_struct *mm)
@@ -191,8 +169,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) }spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); -arm_smmu_tlb_inv_asid (smmu_domain->smmu, smmu_domain->cd.asid); -arm_smmu_atc_inv_domain (smmu_domain,0 ,0 ); +arm_smmu_domain_inv (smmu_domain); }static void arm_smmu_mmu_notifier_free (struct mmu_notifier *mn)
@@ -301,7 +278,7 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)/* * Ensure the ASID is empty in the iommu cache before allowing reuse. */ -arm_smmu_tlb_inv_asid (smmu_domain->smmu, smmu_domain->cd.asid); +arm_smmu_domain_inv (smmu_domain);/* * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -157
211 lines not on this page · raw message
Signature · 1 line
2.43.0