iommu: Allow IOVA rcache range be configured
Subsystems (2)
v3 cover iommu: Allow IOVA rcache range be configured
For streaming DMA mappings involving an IOMMU and whose IOVA len regularly exceeds the IOVA rcache upper limit (meaning that they are not cached), performance can be reduced.
This is much more pronounced from commit 4e89dce72521 ("iommu/iova: Retry from last rb tree node if iova search fails"), as discussed at [0].
IOVAs which cannot be cached are highly involved in the IOVA ageing issue, as discussed at [1].
This series allows the IOVA rcache range be configured, so that we may cache all IOVAs per domain, thus improving performance.
A new IOMMU group sysfs file is added - max_opt_dma_size - which is used indirectly to configure the IOVA rcache range:/sys/kernel/iommu_groups/X/max_opt_dma_size
This file is updated same as how the IOMMU group default domain type is updated, i.e. must unbind the only device in the group first.
The inspiration here comes from block layer request queue sysfs "optimal_io_size" file, in /sys/block/sdX/queue/optimal_io_size
Some figures for storage scenario (when increasing IOVA rcache range to cover all DMA mapping sizes from the LLD):
v5.13-rc1 baseline: 1200K IOPS
With series: 1800K IOPS
All above are for IOMMU strict mode. Non-strict mode gives ~1800K IOPS in all scenarios.
[0] https://lore.kernel.org/linux-iommu/[email protected]/
[1] https://lore.kernel.org/linux-iommu/[email protected]/
Differences to v2:
- Drop DMA mapping API to allow LLD set set for now
- Update default domain immediately, instead of in reprobe
- Fix build warning
Differences to v1:
- Many
- Change method to not operate on a 'live' IOMMU domain:
- rather, force device driver to be re-probed once dma_max_opt_size is set, and reconfig a new IOMMU group then
- Add iommu sysfs max_dma_opt_size file, and allow updating same as how group type is changed
John Garry (7):
iommu: Reactor iommu_group_store_type()
iova: Allow rcache range upper limit to be flexible
iommu: Allow iommu_change_dev_def_domain() realloc default domain for
same type
iova: Add iova_domain_len_is_cached()
iova: Add init_iova_domain_ext()
iommu: Allow max opt DMA len be set for a group via sysfs
dma-iommu: Use init_iova_domain_ext() for IOVA domain init
drivers/iommu/dma-iommu.c | 17 +++- drivers/iommu/iommu.c | 172 ++++++++++++++++++++++++++------------ drivers/iommu/iova.c | 63 +++++++++++--- include/linux/iommu.h | 6 ++ include/linux/iova.h | 21 ++++- 5 files changed, 210 insertions(+), 69 deletions(-)
Signature · 1 line
2.26.2
PATCH v3 1/7 iommu: Reactor iommu_group_store_type()
Function iommu_group_store_type() supports changing the default domain of an IOMMU group.
Many conditions need to be satisfied and steps taken for this action to be successful.
Satisfying these conditions and steps will be required for setting other IOMMU group attributes, so factor into a common part and a part specific to update the IOMMU group attribute.
No functional change intended.
Some code comments are tidied up also.
--- drivers/iommu/iommu.c | 73 +++++++++++++++++++++++++++---------------- 1 file changed, 46 insertions(+), 27 deletions(-)
drivers/iommu/iommu.c +46 -27
@@ -3169,20 +3169,23 @@ static int iommu_change_dev_def_domain(struct iommu_group *group, }/* -* Changing the default domain through sysfs requires the users to ubind the -* drivers from the devices in the iommu group. Return failure if this doesn't -* meet. +* Changing the default domain or any other IOMMU group attribute through sysfs +* requires the users to unbind the drivers from the devices in the IOMMU group. +* Return failure if this precondition is not met. * * We need to consider the race between this and the device release path. * device_lock(dev) is used here to guarantee that the device release path * will not be entered at the same time. */ -static ssize_t iommu_group_store_type (struct iommu_group *group, -const char *buf,size_t count) +static ssize_t iommu_group_store_common (struct iommu_group *group, +const char *buf,size_t count, +int (*cb)(const char *buf, +struct iommu_group *group, +struct device *dev)) {struct group_device *grp_dev;struct device *dev; -int ret, req_type; +int ret;if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO ))return -EACCES ;
@@ -3190,25 +3193,16 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,if (WARN_ON (!group))return -EINVAL ; -if (sysfs_streq (buf,"identity" )) - req_type =IOMMU_DOMAIN_IDENTITY ; -else if (sysfs_streq (buf,"DMA" )) - req_type =IOMMU_DOMAIN_DMA ; -else if (sysfs_streq (buf,"auto" )) - req_type =0 ; -else -return -EINVAL ; -/* * Lock/Unlock the group mutex here before device lock to -* 1. Make sure that the iommu group has only one device (this is a +* 1. Make sure that the IOMMU group has only one device (this is a * prerequisite for step 2) * 2. Get struct *dev which is needed to lock device */ mutex_lock (&group->mutex);if (iommu_group_device_count (group) !=1 ) {mutex_unlock (&group->mutex); -pr_err_ratelimited ("Cannot change default domain: Group has more than one device\n" ); +pr_err_ratelimited ("Cannot change IOMMU group default domain attribute: Group has more than one device\n" );return -EINVAL ; }
@@ -3220,16 +3214,16 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,/* * Don't hold the group mutex because taking group mutex first and then * the device lock could potentially cause a deadlock as below. Assume -* two threads T1 and T2. T1 is trying to change default domain of an -* iommu group and T2 is trying to hot unplug a device or release [1] VF -* of a PCIe device which is in the same iommu group. T1 takes group -* mutex and before it could take device lock assume T2 has taken device -* lock and is yet to take group mutex. Now, both the threads will be -* waiting for the other thread to release lock. Below, lock order was -* suggested. +* two threads, T1 and T2. T1 is trying to change default domain +* attribute of an IOMMU group and T2 is trying to hot unplug a device +* or release [1] VF of a PCIe device which is in the same IOMMU group. +* T1 takes the group mutex and before it could take device lock T2 may +* have taken device lock and is yet to take group mutex. Now, both the +* threads will be waiting for the other thread to release lock. Below, +* lock order was suggested. * device_lock(dev); * mutex_lock(&group->mutex); -* iommu_change_dev_def_domain(); +* cb->iommu_change_dev_def_domain(); [example cb] * mutex_unlock(&group->mutex); * device_unlock(dev); *
@@ -3243,7 +3237,7 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,*/ mutex_unlock (&group->mutex); -/* Check if the device in the group still has a driver bound to it */ +/* Check if the only device in the group still has a driver bound */ device_lock (dev);if (device_is_bound (dev)) {pr_err_ratelimited ("Device is still bound to driver\n" );
@@ -3251,7 +3245,7 @@ static ssize_t iommu_group_store_type(struct iommu_group *group, goto out; } - ret =iommu_change_dev_def_domain (group, dev, req_type); + ret = (cb)(buf, group, dev); ret = ret ?: count; out:
@@ -3260,3 +3254,28 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,return ret; } + +static int iommu_group_store_type_cb (const char *buf, +struct iommu_group *group, +struct device *dev) +{ +int type; + +if (sysfs_streq (buf,"identity" )) + type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + type =0 ; +else +return -EINVAL ; + +return iommu_change_dev_def_domain (group, dev, type); +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) +{ +return iommu_group_store_common (group, buf, count, + iommu_group_store_type_cb); +}
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v3 2/7 iova: Allow rcache range upper limit to be flexible
In reply to John GarrySome LLDs may request DMA mappings whose IOVA length exceeds that of the current rcache upper limit.
This means that allocations for those IOVAs will never be cached, and always must be allocated and freed from the RB tree per DMA mapping cycle. This has a significant affect on performance, more so since commit 4e89dce72521 ("iommu/iova: Retry from last rb tree node if iova search fails"), as discussed at [0].
As a first step towards allowing the rcache range upper limit be configured, hold this value in the IOVA rcache structure, and allocate the rcaches separately.
[0] https://lore.kernel.org/linux-iommu/[email protected]/
--- drivers/iommu/dma-iommu.c | 2 +- drivers/iommu/iova.c | 23 +++++++++++++++++------ include/linux/iova.h | 4 ++-- 3 files changed, 20 insertions(+), 9 deletions(-)
drivers/iommu/dma-iommu.c +1 -1
@@ -432,7 +432,7 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain,* rounding up anything cacheable to make sure that can't happen. The * order of the unadjusted size will still match upon freeing. */ -if (iova_len < (1 << (IOVA_RANGE_CACHE_MAX_SIZE -1 ))) +if (iova_len < (1 << (iovad->rcache_max_size -1 ))) iova_len =roundup_pow_of_two (iova_len); dma_limit =min_not_zero (dma_limit, dev->bus_dma_limit);
drivers/iommu/iova.c +17 -6
@@ -15,6 +15,8 @@/* The anchor node sits above the top of the usable address space */ #define IOVA_ANCHOR ~0UL +#define IOVA_RANGE_CACHE_MAX_SIZE 6/* log of max cached IOVA range size (in pages) */ +static bool iova_rcache_insert (struct iova_domain *iovad,unsigned long pfn,unsigned long size);
@@ -877,7 +879,14 @@ static void init_iova_rcaches(struct iova_domain *iovad)unsigned int cpu;int i; -for (i =0 ; i <IOVA_RANGE_CACHE_MAX_SIZE ; ++i) { + iovad->rcache_max_size =IOVA_RANGE_CACHE_MAX_SIZE ; + + iovad->rcaches =kcalloc (iovad->rcache_max_size, +sizeof (*iovad->rcaches),GFP_KERNEL ); +if (!iovad->rcaches) +return ; + +for (i =0 ; i < iovad->rcache_max_size; ++i) { rcache = &iovad->rcaches[i];spin_lock_init (&rcache->lock); rcache->depot_size =0 ;
@@ -952,7 +961,7 @@ static bool iova_rcache_insert(struct iova_domain *iovad, unsigned long pfn, {unsigned int log_size =order_base_2 (size); -if (log_size >=IOVA_RANGE_CACHE_MAX_SIZE ) +if (log_size >= iovad->rcache_max_size)return false;return __iova_rcache_insert (iovad, &iovad->rcaches[log_size], pfn);
@@ -1008,7 +1017,7 @@ static unsigned long iova_rcache_get(struct iova_domain *iovad, {unsigned int log_size =order_base_2 (size); -if (log_size >=IOVA_RANGE_CACHE_MAX_SIZE ) +if (log_size >= iovad->rcache_max_size)return 0 ;return __iova_rcache_get (&iovad->rcaches[log_size], limit_pfn - size);
@@ -1024,7 +1033,7 @@ static void free_iova_rcaches(struct iova_domain *iovad)unsigned int cpu;int i, j; -for (i =0 ; i <IOVA_RANGE_CACHE_MAX_SIZE ; ++i) { +for (i =0 ; i < iovad->rcache_max_size; ++i) { rcache = &iovad->rcaches[i];for_each_possible_cpu (cpu) { cpu_rcache =per_cpu_ptr (rcache->cpu_rcaches, cpu);
@@ -1035,6 +1044,8 @@ static void free_iova_rcaches(struct iova_domain *iovad)for (j =0 ; j < rcache->depot_size; ++j)iova_magazine_free (rcache->depot[j]); } + +kfree (iovad->rcaches); }/*
@@ -1047,7 +1058,7 @@ static void free_cpu_cached_iovas(unsigned int cpu, struct iova_domain *iovad)unsigned long flags;int i; -for (i =0 ; i <IOVA_RANGE_CACHE_MAX_SIZE ; ++i) { +for (i =0 ; i < iovad->rcache_max_size; ++i) { rcache = &iovad->rcaches[i]; cpu_rcache =per_cpu_ptr (rcache->cpu_rcaches, cpu);spin_lock_irqsave (&cpu_rcache->lock, flags);
@@ -1066,7 +1077,7 @@ static void free_global_cached_iovas(struct iova_domain *iovad)unsigned long flags;int i, j; -for (i =0 ; i <IOVA_RANGE_CACHE_MAX_SIZE ; ++i) { +for (i =0 ; i < iovad->rcache_max_size; ++i) { rcache = &iovad->rcaches[i];spin_lock_irqsave (&rcache->lock, flags);for (j =0 ; j < rcache->depot_size; ++j) {
include/linux/iova.h +2 -2
@@ -25,7 +25,6 @@ struct iova {struct iova_magazine ;struct iova_cpu_rcache ; -#define IOVA_RANGE_CACHE_MAX_SIZE 6/* log of max cached IOVA range size (in pages) */ #define MAX_GLOBAL_MAGS 32/* magazines per bin */ struct iova_rcache {
@@ -74,6 +73,7 @@ struct iova_domain {unsigned long start_pfn;/* Lower limit for this domain */ unsigned long dma_32bit_pfn;unsigned long max32_alloc_size;/* Size of last failed allocation */ +unsigned long rcache_max_size;/* Upper limit of cached IOVA RANGE */ struct iova_fq __percpu *fq;/* Flush Queue */ atomic64_t fq_flush_start_cnt;/* Number of TLB flushes that
@@ -83,7 +83,6 @@ struct iova_domain {have been finished */ struct iova anchor;/* rbtree lookup anchor */ -struct iova_rcache rcaches[IOVA_RANGE_CACHE_MAX_SIZE ];/* IOVA range caches */ iova_flush_cb flush_cb;/* Call-Back function to flush IOMMU TLBs */
@@ -96,6 +95,7 @@ struct iova_domain {atomic_t fq_timer_on;/* 1 when timer is active, 0 when not */ struct hlist_node cpuhp_dead; +struct iova_rcache *rcaches;/* IOVA range caches */ };static inline unsigned long iova_size (struct iova *iova)
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v3 3/7 iommu: Allow iommu_change_dev_def_domain() realloc default domain for same type
In reply to John GarryAllow iommu_change_dev_def_domain() to create a new default domain, keeping the same as current when type is unset.
Also remove comment about function purpose, which will become stale.
--- drivers/iommu/iommu.c | 54 ++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 26 deletions(-)
drivers/iommu/iommu.c +27 -26
@@ -3036,6 +3036,7 @@ u32 iommu_sva_get_pasid(struct iommu_sva *handle) }EXPORT_SYMBOL_GPL (iommu_sva_get_pasid);/* * Changes the default domain of an iommu group that has *only* one device *
@@ -3043,16 +3044,13 @@ EXPORT_SYMBOL_GPL(iommu_sva_get_pasid);* @prev_dev: The device in the group (this is used to make sure that the device * hasn't changed after the caller has called this function) * @type: The type of the new default domain that gets associated with the group +* @new: Allocate new default domain, keeping same type when no type passed * * Returns 0 on success and error code on failure * -* Note: -* 1. Presently, this function is called only when user requests to change the -* group's default domain type through /sys/kernel/iommu_groups/<grp_id>/type -* Please take a closer look if intended to use for other purposes. */ static int iommu_change_dev_def_domain (struct iommu_group *group, -struct device *prev_dev,int type) +struct device *prev_dev,int type,bool new) {struct iommu_domain *prev_dom;struct group_device *grp_dev;
@@ -3105,28 +3103,32 @@ static int iommu_change_dev_def_domain(struct iommu_group *group, goto out; } - dev_def_dom =iommu_get_def_domain_type (dev); -if (!type) { +if (new && !type) { + type = prev_dom->type; + }else { + dev_def_dom =iommu_get_def_domain_type (dev); +if (!type) { +/* +* If the user hasn't requested any specific type of domain and +* if the device supports both the domains, then default to the +* domain the device was booted with +*/ + type = dev_def_dom ? : iommu_def_domain_type; + }else if (dev_def_dom && type != dev_def_dom) { +dev_err_ratelimited (prev_dev,"Device cannot be in %s domain\n" , +iommu_domain_type_str (type)); + ret = -EINVAL ; + goto out; + } +/* -* If the user hasn't requested any specific type of domain and -* if the device supports both the domains, then default to the -* domain the device was booted with +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type */ - type = dev_def_dom ? : iommu_def_domain_type; - }else if (dev_def_dom && type != dev_def_dom) { -dev_err_ratelimited (prev_dev,"Device cannot be in %s domain\n" , -iommu_domain_type_str (type)); - ret = -EINVAL ; - goto out; - } - -/* -* Switch to a new domain only if the requested domain type is different -* from the existing default domain type -*/ -if (prev_dom->type == type) { - ret =0 ; - goto out; +if (prev_dom->type == type) { + ret =0 ; + goto out; + } }/* Sets group->default_domain to the newly allocated domain */
@@ -3270,7 +3272,7 @@ static int iommu_group_store_type_cb(const char *buf,else return -EINVAL ; -return iommu_change_dev_def_domain (group, dev, type); +return iommu_change_dev_def_domain (group, dev, type, false); } static ssize_t iommu_group_store_type(struct iommu_group *group,
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v3 4/7 iova: Add iova_domain_len_is_cached()
In reply to John GarryAdd a function to check whether an IOVA domain currently caches a given upper IOVA len exactly.
--- drivers/iommu/iova.c | 11 +++++++++++ include/linux/iova.h | 8 +++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
drivers/iommu/iova.c +11 -0
@@ -872,6 +872,17 @@ static void iova_magazine_push(struct iova_magazine *mag, unsigned long pfn) mag->pfns[mag->size++] = pfn; } +static unsigned long iova_len_to_rcache_max (unsigned long iova_len) +{ +return order_base_2 (iova_len) +1 ; +} + +/* Test if iova_len range cached upper limit matches that of IOVA domain */ +bool iova_domain_len_is_cached (struct iova_domain *iovad,unsigned long iova_len) +{ +return iova_len_to_rcache_max (iova_len) == iovad->rcache_max_size; +} +static void init_iova_rcaches (struct iova_domain *iovad) {struct iova_cpu_rcache *cpu_rcache;
include/linux/iova.h +7 -1
@@ -136,7 +136,8 @@ static inline unsigned long iova_pfn(struct iova_domain *iovad, dma_addr_t iova)#if IS_ENABLED (CONFIG_IOMMU_IOVA )int iova_cache_get (void );void iova_cache_put (void ); - +bool iova_domain_len_is_cached (struct iova_domain *iovad, +unsigned long iova_len);void free_iova (struct iova_domain *iovad,unsigned long pfn);void __free_iova (struct iova_domain *iovad,struct iova *iova); struct iova *alloc_iova(struct iova_domain *iovad, unsigned long size,
@@ -158,6 +159,11 @@ int init_iova_flush_queue(struct iova_domain *iovad,struct iova *find_iova (struct iova_domain *iovad,unsigned long pfn);void put_iova_domain (struct iova_domain *iovad);#else +static inline bool iova_domain_len_is_cached (struct iova_domain *iovad, +unsigned long iova_len) +{ +return false; +}static inline int iova_cache_get (void ) {return -ENOTSUPP ;
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v3 5/7 iova: Add init_iova_domain_ext()
In reply to John GarryAdd extended version of init_iova_domain() which accepts an max opt iova length argument, and use it to set the rcaches range.
This can be combined with the normal version later.
--- drivers/iommu/iova.c | 31 ++++++++++++++++++++++++------- include/linux/iova.h | 9 +++++++++ 2 files changed, 33 insertions(+), 7 deletions(-)
drivers/iommu/iova.c +24 -7
@@ -23,7 +23,7 @@ static bool iova_rcache_insert(struct iova_domain *iovad,static unsigned long iova_rcache_get (struct iova_domain *iovad,unsigned long size,unsigned long limit_pfn); -static void init_iova_rcaches (struct iova_domain *iovad); +static void init_iova_rcaches (struct iova_domain *iovad,unsigned long iova_len);static void free_cpu_cached_iovas (unsigned int cpu,struct iova_domain *iovad);static void free_iova_rcaches (struct iova_domain *iovad);static void fq_destroy_all_entries (struct iova_domain *iovad);
@@ -46,9 +46,9 @@ static struct iova *to_iova(struct rb_node *node) return rb_entry(node, struct iova, node); } -void -init_iova_domain (struct iova_domain *iovad,unsigned long granule, -unsigned long start_pfn) +static void +__init_iova_domain (struct iova_domain *iovad,unsigned long granule, +unsigned long start_pfn,unsigned long iova_len) {/* * IOVA granularity will normally be equal to the smallest
@@ -71,7 +71,21 @@ init_iova_domain(struct iova_domain *iovad, unsigned long granule,rb_link_node (&iovad->anchor.node,NULL , &iovad->rbroot.rb_node);rb_insert_color (&iovad->anchor.node, &iovad->rbroot);cpuhp_state_add_instance_nocalls (CPUHP_IOMMU_IOVA_DEAD , &iovad->cpuhp_dead); -init_iova_rcaches (iovad); +init_iova_rcaches (iovad, iova_len); +} + +void +init_iova_domain_ext (struct iova_domain *iovad,unsigned long granule, +unsigned long start_pfn,unsigned long iova_len) +{ +__init_iova_domain (iovad, granule, start_pfn, iova_len); +} + +void +init_iova_domain (struct iova_domain *iovad,unsigned long granule, +unsigned long start_pfn) +{ +__init_iova_domain (iovad, granule, start_pfn,0 ); }EXPORT_SYMBOL_GPL (init_iova_domain);
@@ -883,14 +897,17 @@ bool iova_domain_len_is_cached(struct iova_domain *iovad, unsigned long iova_lenreturn iova_len_to_rcache_max (iova_len) == iovad->rcache_max_size; } -static void init_iova_rcaches (struct iova_domain *iovad) +static void init_iova_rcaches (struct iova_domain *iovad,unsigned long iova_len) {struct iova_cpu_rcache *cpu_rcache;struct iova_rcache *rcache;unsigned int cpu;int i; - iovad->rcache_max_size =IOVA_RANGE_CACHE_MAX_SIZE ; +if (iova_len) + iovad->rcache_max_size =iova_len_to_rcache_max (iova_len); +else + iovad->rcache_max_size =IOVA_RANGE_CACHE_MAX_SIZE ; iovad->rcaches =kcalloc (iovad->rcache_max_size,sizeof (*iovad->rcaches),GFP_KERNEL );
include/linux/iova.h +9 -0
@@ -154,6 +154,8 @@ struct iova *reserve_iova(struct iova_domain *iovad, unsigned long pfn_lo, unsigned long pfn_hi);void init_iova_domain (struct iova_domain *iovad,unsigned long granule,unsigned long start_pfn); +void init_iova_domain_ext (struct iova_domain *iovad,unsigned long granule, +unsigned long start_pfn,unsigned long iova_len);int init_iova_flush_queue (struct iova_domain *iovad,iova_flush_cb flush_cb,iova_entry_dtor entry_dtor);struct iova *find_iova (struct iova_domain *iovad,unsigned long pfn);
@@ -222,6 +224,13 @@ static inline void init_iova_domain(struct iova_domain *iovad, { } +static inline void init_iova_domain_ext (struct iova_domain *iovad, +unsigned long granule, +unsigned long start_pfn, +unsigned long iova_len) +{ +} +static inline int init_iova_flush_queue (struct iova_domain *iovad,iova_flush_cb flush_cb,iova_entry_dtor entry_dtor)
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v3 6/7 iommu: Allow max opt DMA len be set for a group via sysfs
In reply to John GarryAdd support to allow the maximum optimised DMA len be set for an IOMMU group via sysfs.
This much the same with the method to change the default domain type for a group.
--- drivers/iommu/iommu.c | 51 +++++++++++++++++++++++++++++++++++++++++-- include/linux/iommu.h | 6 +++++ 2 files changed, 55 insertions(+), 2 deletions(-)
drivers/iommu/iommu.c +49 -2
@@ -45,6 +45,7 @@ struct iommu_group {struct iommu_domain *default_domain;struct iommu_domain *domain;struct list_head entry; +size_t max_opt_dma_size; };struct group_device {
@@ -86,6 +87,9 @@ static int iommu_create_device_direct_mappings(struct iommu_group *group,static struct iommu_group *iommu_group_get_for_dev (struct device *dev);static ssize_t iommu_group_store_type (struct iommu_group *group,const char *buf,size_t count); +static ssize_t iommu_group_store_max_opt_dma_size (struct iommu_group *group, +const char *buf, +size_t count);#define IOMMU_GROUP_ATTR (_name, _mode, _show, _store) \ struct iommu_group_attribute iommu_group_attr_##_name = \
@@ -554,6 +558,12 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,return strlen (type); } +static ssize_t iommu_group_show_max_opt_dma_size (struct iommu_group *group, +char *buf) +{ +return sprintf (buf,"%zu\n" , group->max_opt_dma_size); +} + static IOMMU_GROUP_ATTR(name, S_IRUGO, iommu_group_show_name, NULL); static IOMMU_GROUP_ATTR(reserved_regions, 0444,
@@ -562,6 +572,9 @@ static IOMMU_GROUP_ATTR(reserved_regions, 0444, static IOMMU_GROUP_ATTR(type, 0644, iommu_group_show_type, iommu_group_store_type); +static IOMMU_GROUP_ATTR(max_opt_dma_size, 0644, iommu_group_show_max_opt_dma_size, + iommu_group_store_max_opt_dma_size); +static void iommu_group_release (struct kobject *kobj) {struct iommu_group *group =to_iommu_group (kobj);
@@ -648,6 +661,10 @@ struct iommu_group *iommu_group_alloc(void)if (ret)return ERR_PTR (ret); + ret =iommu_group_create_file (group, &iommu_group_attr_max_opt_dma_size); +if (ret) +return ERR_PTR (ret); +pr_debug ("Allocated group %d\n" , group->id);return group;
@@ -2279,6 +2296,11 @@ struct iommu_domain *iommu_get_dma_domain(struct device *dev)return dev->iommu_group->default_domain; } +size_t iommu_group_get_max_opt_dma_size (struct iommu_group *group) +{ +return group->max_opt_dma_size; +} +/* * IOMMU groups are really the natural working unit of the IOMMU, but * the IOMMU API works on domains and devices. Bridge that gap by
@@ -3045,12 +3067,14 @@ EXPORT_SYMBOL_GPL(iommu_sva_get_pasid);* hasn't changed after the caller has called this function) * @type: The type of the new default domain that gets associated with the group * @new: Allocate new default domain, keeping same type when no type passed +* @max_opt_dma_size: If set, set the IOMMU group max_opt_dma_size when success * * Returns 0 on success and error code on failure * */ static int iommu_change_dev_def_domain (struct iommu_group *group, -struct device *prev_dev,int type,bool new) +struct device *prev_dev,int type,bool new, +unsigned long max_opt_dma_size) {struct iommu_domain *prev_dom;struct group_device *grp_dev;
@@ -3146,6 +3170,9 @@ static int iommu_change_dev_def_domain(struct iommu_group *group, group->domain = group->default_domain; +if (max_opt_dma_size) + group->max_opt_dma_size = max_opt_dma_size; +/* * Release the mutex here because ops->probe_finalize() call-back of * some vendor IOMMU drivers calls arm_iommu_attach_device() which
@@ -3272,7 +3299,7 @@ static int iommu_group_store_type_cb(const char *buf,else return -EINVAL ; -return iommu_change_dev_def_domain (group, dev, type, false); +return iommu_change_dev_def_domain (group, dev, type, false,0 ); } static ssize_t iommu_group_store_type(struct iommu_group *group,
@@ -3281,3 +3308,23 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,return iommu_group_store_common (group, buf, count, iommu_group_store_type_cb); } + +static int iommu_group_store_max_opt_dma_size_cb (const char *buf, +struct iommu_group *group, +struct device *dev) +{ +unsigned long val; + +if (kstrtoul (buf,0 , &val) || !val) +return -EINVAL ; + +return iommu_change_dev_def_domain (group, dev,0 , true, val); +} + +static ssize_t iommu_group_store_max_opt_dma_size (struct iommu_group *group, +const char *buf, +size_t count) +{ +return iommu_group_store_common (group, buf, count, + iommu_group_store_max_opt_dma_size_cb); +}
include/linux/iommu.h +6 -0
@@ -405,6 +405,7 @@ extern int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid);extern struct iommu_domain *iommu_get_domain_for_dev (struct device *dev);extern struct iommu_domain *iommu_get_dma_domain (struct device *dev); +extern size_t iommu_group_get_max_opt_dma_size (struct iommu_group *group);extern int iommu_map (struct iommu_domain *domain,unsigned long iova,phys_addr_t paddr,size_t size,int prot); extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova,
@@ -653,6 +654,11 @@ static inline struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)return NULL ; } +static inline size_t iommu_group_get_max_opt_dma_size (struct iommu_group *group) +{ +return 0 ; +} +static inline int iommu_map (struct iommu_domain *domain,unsigned long iova,phys_addr_t paddr,size_t size,int prot) {
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v3 7/7 dma-iommu: Use init_iova_domain_ext() for IOVA domain init
In reply to John GarryPass the max opt iova len to init the IOVA domain, if set.
--- drivers/iommu/dma-iommu.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
drivers/iommu/dma-iommu.c +14 -1
@@ -333,6 +333,8 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,struct iommu_dma_cookie *cookie = domain->iova_cookie;unsigned long order, base_pfn;struct iova_domain *iovad; +size_t max_opt_dma_size; +unsigned long iova_len;if (!cookie || cookie->type !=IOMMU_DMA_IOVA_COOKIE )return -EINVAL ;
@@ -366,7 +368,18 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,return 0 ; } -init_iova_domain (iovad,1UL << order, base_pfn); + max_opt_dma_size =iommu_group_get_max_opt_dma_size (dev->iommu_group); + +if (max_opt_dma_size) { +unsigned long shift =__ffs (1UL << order); + + iova_len = max_opt_dma_size >> shift; + iova_len =roundup_pow_of_two (iova_len); + }else { + iova_len =0 ; + } + +init_iova_domain_ext (iovad,1UL << order, base_pfn, iova_len);if (!cookie->fq_domain && (!dev || !dev_is_untrusted (dev)) && domain->ops->flush_iotlb_all && !iommu_get_dma_strict (domain)) {
Signature · 6 lines
2.26.2 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 13 lines · John Garry
For streaming DMA mappings involving an IOMMU and whose IOVA len regularly exceeds the IOVA rcache upper limit (meaning that they are not cached), performance can be reduced.
This is much more pronounced from commit
4e89dce72521("iommu/iova: Retry from last rb tree node if iova search fails"), as discussed at [0].IOVAs which cannot be cached are highly involved in the IOVA ageing issue, as discussed at [1].
This series allows the IOVA rcache range be configured, so that we may cache all IOVAs per domain, thus improving performance.
Quoted 6 lines
A new IOMMU group sysfs file is added -
max_opt_dma_size- which is used indirectly to configure the IOVA rcache range:/sys/kernel/iommu_groups/X/max_opt_dma_sizeThis file is updated same as how the IOMMU group default domain type is updated, i.e. must unbind the only device in the group first.
Could you explain why it requires singleton group and driver unbinding if the user only wants to increase the upper limit? I haven't dived into the details yet, sorry if this is a silly question.
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 18 lines · Lu Baolu
Quoted 17 lines · John Garry
For streaming DMA mappings involving an IOMMU and whose IOVA len regularly exceeds the IOVA rcache upper limit (meaning that they are not cached), performance can be reduced.
This is much more pronounced from commit
4e89dce72521("iommu/iova: Retry from last rb tree node if iova search fails"), as discussed at [0].IOVAs which cannot be cached are highly involved in the IOVA ageing issue, as discussed at [1].
This series allows the IOVA rcache range be configured, so that we may cache all IOVAs per domain, thus improving performance.
A new IOMMU group sysfs file is added -
max_opt_dma_size- which is used indirectly to configure the IOVA rcache range:/sys/kernel/iommu_groups/X/max_opt_dma_size
Quoted 6 lines
Quoted 2 lines
This file is updated same as how the IOMMU group default domain type is updated, i.e. must unbind the only device in the group first.
Could you explain why it requires singleton group and driver unbinding if the user only wants to increase the upper limit? I haven't dived into the details yet, sorry if this is a silly question.
Hi Baolu,
I did actually try increasing the range for a 'live' domain in the v1 series, but it turned out too messy. First problem is reallocating the memory to hold the rcaches. Second problem is that we need to deal with the issue that all IOVAs in the rcache need to be a pow-of-2, which is difficult to enforce for IOVAs which weren't being cached before, but now would be.
So now I changed to work similar to how we change the default domain type, i.e. don't operate on a 'live' domain.
Thanks,
John
Quoted 26 lines · John Garry
Quoted 18 lines · Lu Baolu
Quoted 17 lines · John Garry
For streaming DMA mappings involving an IOMMU and whose IOVA len regularly exceeds the IOVA rcache upper limit (meaning that they are not cached), performance can be reduced.
This is much more pronounced from commit
4e89dce72521("iommu/iova: Retry from last rb tree node if iova search fails"), as discussed at [0].IOVAs which cannot be cached are highly involved in the IOVA ageing issue, as discussed at [1].
This series allows the IOVA rcache range be configured, so that we may cache all IOVAs per domain, thus improving performance.
A new IOMMU group sysfs file is added -
max_opt_dma_size- which is used indirectly to configure the IOVA rcache range:/sys/kernel/iommu_groups/X/max_opt_dma_sizeQuoted 6 lines
Quoted 2 lines
This file is updated same as how the IOMMU group default domain type is updated, i.e. must unbind the only device in the group first.
Could you explain why it requires singleton group and driver unbinding if the user only wants to increase the upper limit? I haven't dived into the details yet, sorry if this is a silly question.
Quoted 5 lines
Hi Baolu,
I did actually try increasing the range for a 'live' domain in the v1 series, but it turned out too messy. First problem is reallocating the memory to hold the rcaches. Second problem is that we need to deal with the issue that all IOVAs in the rcache need to be a pow-of-2, which is difficult to enforce for IOVAs which weren't being cached before, but now would be.
So now I changed to work similar to how we change the default domain type, i.e. don't operate on a 'live' domain.
If these hard restrictions on users are just to walk around the messy code in kernel, I would rather solve those messy code to achieve a better user experience. :-)
Best regards,
baolu
Quoted 7 lines · Lu Baolu
Quoted 3 lines
I did actually try increasing the range for a 'live' domain in the v1 series, but it turned out too messy. First problem is reallocating the memory to hold the rcaches. Second problem is that we need to deal with the issue that all IOVAs in the rcache need to be a pow-of-2, which is difficult to enforce for IOVAs which weren't being cached before, but now would be.
So now I changed to work similar to how we change the default domain type, i.e. don't operate on a 'live' domain.
If these hard restrictions on users are just to walk around the messy code in kernel, I would rather solve those messy code to achieve a better user experience. :-)
Hi Baolu,
It's not that the code is messy. I'm just saying that it's difficult to change the rcache range of a live domain. So we take an approach similar to changing the domain default type - it is more straightforward to configure the rcache range that way.
Anyway, it would be great if you or anyone could look at the code so that some progress can be made here.
Thanks,
John