User API for nested shared virtual address (SVA)
cover User API for nested shared virtual address (SVA)
This set consists of IOMMU APIs to support SVA in the guest, a.k.a nested SVA. As the complete SVA support is complex, we break down the enabling effort into three stages:
1. PCI device direct assignment
2. Fault handling, especially page request service support
3. Mediated device assignment
Each stage includes common API and vendor specific IOMMU driver changes. This series is the common uAPI for stage #1. It is intended to build consensus on the interface which all vendors reply on.
This series is extracted from the complete stage1 set which includes VT-d code.
https://lkml.org/lkml/2019/8/15/951
Changes:
- Use spinlock instead of mutex to protect ioasid custom allocators. This is to support callers in atomic context
- Added more padding to guest PASID bind data for future extensions, suggested by Joerg.
After much thinking, I did not do name change from PASID to IOASID in the uAPI, considering we have been using PASID in the rest of uAPIs. IOASID will remain used within the kernel.
For more discussions lead to this series, checkout LPC 2019 VFIO/IOMMU/PCI microconference materials.
https://linuxplumbersconf.org/event/4/sessions/66/#20190909
Jacob Pan (2):
iommu/ioasid: Add custom allocators
iommu: Introduce guest PASID bind function
Jean-Philippe Brucker (1):
iommu: Add I/O ASID allocator
Yi L Liu (1):
iommu: Introduce cache_invalidate API
drivers/iommu/Kconfig | 4 + drivers/iommu/Makefile | 1 + drivers/iommu/ioasid.c | 432 +++++++++++++++++++++++++++++++++++++++++++++ drivers/iommu/iommu.c | 30 ++++ include/linux/ioasid.h | 75 ++++++++ include/linux/iommu.h | 36 ++++ include/uapi/linux/iommu.h | 169 ++++++++++++++++++ 7 files changed, 747 insertions(+) create mode 100644 drivers/iommu/ioasid.c create mode 100644 include/linux/ioasid.h
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 1/4 iommu: Introduce cache_invalidate API
From: Yi L Liu <[email protected]>
In any virtualization use case, when the first translation stage is "owned" by the guest OS, the host IOMMU driver has no knowledge of caching structure updates unless the guest invalidation activities are trapped by the virtualizer and passed down to the host.
Since the invalidation data can be obtained from user space and will be written into physical IOMMU, we must allow security check at various layers. Therefore, generic invalidation data format are proposed here, model specific IOMMU drivers need to convert them into their own format.
--- drivers/iommu/iommu.c | 10 +++++ include/linux/iommu.h | 14 ++++++ include/uapi/linux/iommu.h | 110 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 134 insertions(+)
drivers/iommu/iommu.c +10 -0
@@ -1610,6 +1610,16 @@ int iommu_attach_device(struct iommu_domain *domain, struct device *dev) }EXPORT_SYMBOL_GPL (iommu_attach_device); +int iommu_cache_invalidate (struct iommu_domain *domain,struct device *dev, +struct iommu_cache_invalidate_info *inv_info) +{ +if (unlikely (!domain->ops->cache_invalidate)) +return -ENODEV ; + +return domain->ops->cache_invalidate (domain, dev, inv_info); +} +EXPORT_SYMBOL_GPL (iommu_cache_invalidate); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {
include/linux/iommu.h +14 -0
@@ -228,6 +228,7 @@ struct iommu_sva_ops {* @sva_unbind: Unbind process address space from device * @sva_get_pasid: Get PASID associated to a SVA handle * @page_response: handle page request response +* @cache_invalidate: invalidate translation caches * @pgsize_bitmap: bitmap of all possible supported page sizes */ struct iommu_ops {
@@ -291,6 +292,8 @@ struct iommu_ops {int (*page_response)(struct device *dev,struct iommu_fault_event *evt,struct iommu_page_response *msg); +int (*cache_invalidate)(struct iommu_domain *domain,struct device *dev, +struct iommu_cache_invalidate_info *inv_info);unsigned long pgsize_bitmap; };
@@ -395,6 +398,9 @@ extern int iommu_attach_device(struct iommu_domain *domain, struct device *dev);extern void iommu_detach_device (struct iommu_domain *domain,struct device *dev); +extern int iommu_cache_invalidate (struct iommu_domain *domain, +struct device *dev, +struct iommu_cache_invalidate_info *inv_info);extern struct iommu_domain *iommu_get_domain_for_dev (struct device *dev);extern struct iommu_domain *iommu_get_dma_domain (struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
@@ -937,6 +943,14 @@ static inline int iommu_sva_get_pasid(struct iommu_sva *handle) return IOMMU_PASID_INVALID; } +static inline int +iommu_cache_invalidate (struct iommu_domain *domain, +struct device *dev, +struct iommu_cache_invalidate_info *inv_info) +{ +return -ENODEV ; +} +#endif /* CONFIG_IOMMU_API */ #ifdef CONFIG_IOMMU_DEBUGFS
include/uapi/linux/iommu.h +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH 1/4] iommu: Introduce cache_invalidate API
Quoted 12 lines · Jacob Pan
From: Yi L Liu <[email protected]>
In any virtualization use case, when the first translation stage is "owned" by the guest OS, the host IOMMU driver has no knowledge of caching structure updates unless the guest invalidation activities are trapped by the virtualizer and passed down to the host.
Since the invalidation data can be obtained from user space and will be written into physical IOMMU, we must allow security check at various layers. Therefore, generic invalidation data format are proposed here, model specific IOMMU drivers need to convert them into their own format.
Quoted 5 lines
Signed-off-by Yi L Liu <[email protected]>Signed-off-by Jacob Pan <[email protected]>Signed-off-by Ashok Raj <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>
I tried to take a fresh look at this and didn't see anything problematic, though I might have been the last one to edit it. Anyway:
PATCH 2/4 iommu: Add I/O ASID allocator
In reply to Jacob PanFrom: Jean-Philippe Brucker <[email protected]>
Some devices might support multiple DMA address spaces, in particular those that have the PCI PASID feature. PASID (Process Address Space ID) allows to share process address spaces with devices (SVA), partition a device into VM-assignable entities (VFIO mdev) or simply provide multiple DMA address space to kernel drivers. Add a global PASID allocator usable by different drivers at the same time. Name it I/O ASID to avoid confusion with ASIDs allocated by arch code, which are usually a separate ID space.
The IOASID space is global. Each device can have its own PASID space, but by convention the IOMMU ended up having a global PASID space, so that with SVA, each mm_struct is associated to a single PASID.
The allocator is primarily used by IOMMU subsystem but in rare occasions drivers would like to allocate PASIDs for devices that aren't managed by an IOMMU, using the same ID space as IOMMU.
--- drivers/iommu/Kconfig | 4 ++ drivers/iommu/Makefile | 1 + drivers/iommu/ioasid.c | 151 +++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/ioasid.h | 47 +++++++++++++++ 4 files changed, 203 insertions(+) create mode 100644 drivers/iommu/ioasid.c create mode 100644 include/linux/ioasid.h
drivers/iommu/Kconfig +4 -0
@@ -3,6 +3,10 @@ config IOMMU_IOVA tristate +# The IOASID library may also be used by non-IOMMU_API users +config IOASID + tristate + # IOMMU_API always gets selected by whoever wants it. config IOMMU_API bool
drivers/iommu/Makefile +1 -0
@@ -7,6 +7,7 @@ obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o +obj-$(CONFIG_IOASID) += ioasid.o obj-$(CONFIG_IOMMU_IOVA) += iova.o obj-$(CONFIG_OF_IOMMU) += of_iommu.o obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o
drivers/iommu/ioasid.c new file +151 -0
151 lines not on this page · raw message
include/linux/ioasid.h new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.7.4
kbuild test robot
build botRe: [PATCH 2/4] iommu: Add I/O ASID allocator
I love your patch! Yet something to improve:
Hi Jacob,
I love your patch! Yet something to improve:
[auto build test ERROR on linus/master]
[cannot apply to v5.3 next-20190918]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Jacob-Pan/iommu-Introduce-cache_invalidate-API/20190919-072517
config: ia64-allmodconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=ia64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from <command-line>:0:0:
include/linux/ioasid.h: In function 'ioasid_set_data':
>> include/linux/ioasid.h:43:10: error: 'ENODEV' undeclared (first use in this function)
return -ENODEV;
^~~~~~ include/linux/ioasid.h:43:10: note: each undeclared identifier is reported only once for each function it appears in
vim +/ENODEV +43 include/linux/ioasid.h
40
41 static inline int ioasid_set_data(ioasid_t ioasid, void *data)
42 {
> 43 return -ENODEV;
44 }
45 ---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH 2/4] iommu: Add I/O ASID allocator
In reply to Jacob PanQuoted 15 lines · Jacob Pan
From: Jean-Philippe Brucker <[email protected]>
Some devices might support multiple DMA address spaces, in particular those that have the PCI PASID feature. PASID (Process Address Space ID) allows to share process address spaces with devices (SVA), partition a device into VM-assignable entities (VFIO mdev) or simply provide multiple DMA address space to kernel drivers. Add a global PASID allocator usable by different drivers at the same time. Name it I/O ASID to avoid confusion with ASIDs allocated by arch code, which are usually a separate ID space.
The IOASID space is global. Each device can have its own PASID space, but by convention the IOMMU ended up having a global PASID space, so that with SVA, each
mm_structis associated to a single PASID.
Quoted 6 lines
The allocator is primarily used by IOMMU subsystem but in rare occasions drivers would like to allocate PASIDs for devices that aren't managed by an IOMMU, using the same ID space as IOMMU.
Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Jacob Pan <[email protected]>
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 3/4 iommu/ioasid: Add custom allocators
In reply to Jacob PanIOASID allocation may rely on platform specific methods. One use case is that when running in the guest, in order to obtain system wide global IOASIDs, emulated allocation interface is needed to communicate with the host. Here we call these platform specific allocators custom allocators.
Custom IOASID allocators can be registered at runtime and take precedence over the default XArray allocator. They have these attributes:
- provides platform specific alloc()/free() functions with private data.
- allocation results lookup are not provided by the allocator, lookup request must be done by the IOASID framework by its own XArray.
- allocators can be unregistered at runtime, either fallback to the next custom allocator or to the default allocator.
- custom allocators can share the same set of alloc()/free() helpers, in this case they also share the same IOASID space, thus the same XArray.
- switching between allocators requires all outstanding IOASIDs to be freed unless the two allocators share the same alloc()/free() helpers.
--- drivers/iommu/ioasid.c | 301 +++++++++++++++++++++++++++++++++++++++++++++++-- include/linux/ioasid.h | 28 +++++ 2 files changed, 319 insertions(+), 10 deletions(-)
drivers/iommu/ioasid.c +291 -10
@@ -17,7 +17,254 @@ struct ioasid_data {struct rcu_head rcu; }; -static DEFINE_XARRAY_ALLOC (ioasid_xa); +/* +* struct ioasid_allocator_data - Internal data structure to hold information +* about an allocator. There are two types of allocators: +* +* - Default allocator always has its own XArray to track the IOASIDs allocated. +* - Custom allocators may share allocation helpers with different private data. +* Custom allocators share the same helper functions also share the same +* XArray. +* Rules: +* 1. Default allocator is always available, not dynamically registered. This is +* to prevent race conditions with early boot code that want to register +* custom allocators or allocate IOASIDs. +* 2. Custom allocators take precedence over the default allocator. +* 3. When all custom allocators sharing the same helper functions are +* unregistered (e.g. due to hotplug), all outstanding IOASIDs must be +* freed. +* 4. When switching between custom allocators sharing the same helper +* functions, outstanding IOASIDs are preserved. +* 5. When switching between custom allocator and default allocator, all IOASIDs +* must be freed to ensure unadulterated space for the new allocator. +* +* @ops: allocator helper functions and its data +* @list: registered custom allocators +* @slist: allocators share the same ops but different data +* @flags: attributes of the allocator +* @xa xarray holds the IOASID space +* @users number of allocators sharing the same ops and XArray +*/ +struct ioasid_allocator_data { +struct ioasid_allocator_ops *ops; +struct list_head list; +struct list_head slist; +#define IOASID_ALLOCATOR_CUSTOM BIT(0)/* Needs framework to track results */ +unsigned long flags; +struct xarray xa; +refcount_t users; +}; + +static DEFINE_SPINLOCK (ioasid_allocator_lock); +static LIST_HEAD (allocators_list); + +static ioasid_t default_alloc (ioasid_t min,ioasid_t max,void *opaque); +static void default_free (ioasid_t ioasid,void *opaque); + +static struct ioasid_allocator_ops default_ops = { + .alloc = default_alloc, + .free = default_free, +}; + +static struct ioasid_allocator_data default_allocator = { + .ops = &default_ops, + .flags =0 , + .xa =XARRAY_INIT (ioasid_xa,XA_FLAGS_ALLOC ), +}; + +static struct ioasid_allocator_data *active_allocator = &default_allocator; + +static ioasid_t default_alloc (ioasid_t min,ioasid_t max,void *opaque) +{ +ioasid_t id; + +if (xa_alloc (&default_allocator.xa, &id, opaque,XA_LIMIT (min, max),GFP_ATOMIC )) { +pr_err ("Failed to alloc ioasid from %d to %d\n" , min, max); +return INVALID_IOASID ; + } + +return id; +} + +static void default_free (ioasid_t ioasid,void *opaque) +{ +struct ioasid_data *ioasid_data; + + ioasid_data =xa_erase (&default_allocator.xa, ioasid); +kfree_rcu (ioasid_data, rcu); +} + +/* Allocate and initialize a new custom allocator with its helper functions */ +static struct ioasid_allocator_data *ioasid_alloc_allocator (struct ioasid_allocator_ops *ops) +{ +struct ioasid_allocator_data *ia_data; + + ia_data =kzalloc (sizeof (*ia_data),GFP_ATOMIC ); +if (!ia_data) +return NULL ; + +xa_init_flags (&ia_data->xa,XA_FLAGS_ALLOC ); +INIT_LIST_HEAD (&ia_data->slist); + ia_data->flags |=IOASID_ALLOCATOR_CUSTOM ; + ia_data->ops = ops; + +/* For tracking custom allocators that share the same ops */ +list_add_tail (&ops->list, &ia_data->slist); +refcount_set (&ia_data->users,1 ); + +return ia_data; +} + +static bool use_same_ops (struct ioasid_allocator_ops *a,struct ioasid_allocator_ops *b) +{ +return (a->free == b->free) && (a->alloc == b->alloc); +} + +/** +* ioasid_register_allocator - register a custom allocator +* @ops: the custom allocator ops to be registered +* +* Custom allocators take precedence over the default xarray based allocator. +* Private data associated with the IOASID allocated by the custom allocators +* are managed by IOASID framework similar to data stored in xa by default +* allocator. +* +* There can be multiple allocators registered but only one is active. In case +* of runtime removal of a custom allocator, the next one is activated based +* on the registration ordering. +* +* Multiple allocators can share the same alloc() function, in this case the +* IOASID space is shared. +*/ +int ioasid_register_allocator (struct ioasid_allocator_ops *ops) +{ +struct ioasid_allocator_data *ia_data; +struct ioasid_allocator_data *pallocator; +int ret =0 ; + +spin_lock (&ioasid_allocator_lock); + + ia_data =ioasid_alloc_allocator (ops); +if (!ia_data) { + ret = -ENOMEM ; + goto out_unlock; + } + +/* +* No particular preference, we activate the first one and keep +* the later registered allocators in a list in case the first one gets +* removed due to hotplug. +*/ +if (list_empty (&allocators_list)) { +WARN_ON (active_allocator != &default_allocator); +/* Use this new allocator if default is not active */ +if (xa_empty (&active_allocator->xa)) { + active_allocator = ia_data; +list_add_tail (&ia_data->list, &allocators_list); + goto out_unlock; + } +pr_warn ("Default allocator active with outstanding IOASID\n" ); + ret = -EAGAIN ; + goto out_free; + } + +/* Check if the allocator is already registered */ +list_for_each_entry (pallocator, &allocators_list, list) { +if (pallocator->ops == ops) { +pr_err ("IOASID allocator already registered\n" ); + ret = -EEXIST ; + goto out_free; + }else if (use_same_ops (pallocator->ops, ops)) { +/* +* If the new allocator shares the same ops, +* then they will share the same IOASID space. +* We should put them under the same xarray. +*/ +list_add_tail (&ops->list, &pallocator->slist); +refcount_inc (&pallocator->users); +pr_info ("New IOASID allocator ops shared %u times\n" , +refcount_read (&pallocator->users)); + goto out_free; + } + } +list_add_tail (&ia_data->list, &allocators_list); + +spin_unlock (&ioasid_allocator_lock); +return 0 ; +out_free: +kfree (ia_data); +out_unlock: +spin_unlock (&ioasid_allocator_lock); +return ret; +} +EXPORT_SYMBOL_GPL (ioasid_register_allocator); + +static inline bool has_shared_ops (struct ioasid_allocator_data *allocator) +{ +return refcount_read (&allocator->users) >1 ; +} + +/** +* ioasid_unregister_allocator - Remove a custom IOASID allocator ops +* @ops: the custom allocator to be removed +* +* Remove an allocator from the list, activate the next allocator in +* the order it was registered. Or revert to default allocator if all +* custom allocators are unregistered without outstanding IOASIDs. +*/ +void ioasid_unregister_allocator (struct ioasid_allocator_ops *ops) +{ +struct ioasid_allocator_data *pallocator; +struct ioasid_allocator_ops *sops; + +spin_lock (&ioasid_allocator_lock); +if (list_empty (&allocators_list)) { +pr_warn ("No custom IOASID allocators active!\n" ); + goto exit_unlock; + } + +list_for_each_entry (pallocator, &allocators_list, list) { +if (!use_same_ops (pallocator->ops, ops)) +continue ; + +if (refcount_read (&pallocator->users) ==1 ) { +/* No shared helper functions */ +list_del (&pallocator->list); +/* +* All IOASIDs should have been freed before +* the last allocator that shares the same ops +* is unregistered. +*/ +WARN_ON (!xa_empty (&pallocator->xa)); +kfree (pallocator); +if (list_empty (&allocators_list)) { +pr_info ("No custom IOASID allocators, switch to default.\n" ); + active_allocator = &default_allocator; + }else if (pallocator == active_allocator) { + active_allocator = list_first_entry(&allocators_list, struct ioasid_allocator_data, list); +pr_info ("IOASID allocator changed" ); + } +break ; + } +/* +* Find the matching shared ops to delete, +* but keep outstanding IOASIDs +*/ +list_for_each_entry (sops, &pallocator->slist, list) { +if (sops == ops) { +list_del (&ops->list); +if (refcount_dec_and_test (&pallocator->users)) +pr_err ("no shared ops\n" ); +break ; + } + } +break ; + } + +exit_unlock: +spin_unlock (&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL (ioasid_unregister_allocator);/** * ioasid_set_data - Set private data for an allocated ioasid
@@ -32,13 +279,13 @@ int ioasid_set_data(ioasid_t ioasid, void *data)struct ioasid_data *ioasid_data;int ret =0 ; -xa_lock (&ioasid_xa); - ioasid_data =xa_load (&ioasid_xa, ioasid); +spin_lock (&ioasid_allocator_lock); + ioasid_data =xa_load (&active_allocator->xa, ioasid);if (ioasid_data)rcu_assign_pointer (ioasid_data->private, data);else ret = -ENOENT ; -xa_unlock (&ioasid_xa); +spin_unlock (&ioasid_allocator_lock);/* * Wait for readers to stop accessing the old private data, so the
@@ -66,8 +313,9 @@ EXPORT_SYMBOL_GPL(ioasid_set_data);ioasid_t ioasid_alloc (struct ioasid_set *set,ioasid_t min,ioasid_t max,void *private) { -ioasid_t id;struct ioasid_data *data; +void *adata; +ioasid_t id; data =kzalloc (sizeof (*data),GFP_KERNEL );if (!data)
@@ -76,14 +324,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; -if (xa_alloc (&ioasid_xa, &id, data,XA_LIMIT (min, max),GFP_KERNEL )) { -pr_err ("Failed to alloc ioasid from %d to %d\n" , min, max); +/* +* Custom allocator needs allocator data to perform platform specific +* operations. +*/ +spin_lock (&ioasid_allocator_lock); + adata = active_allocator->flags &IOASID_ALLOCATOR_CUSTOM ? active_allocator->ops->pdata : data; + id = active_allocator->ops->alloc (min, max, adata); +if (id ==INVALID_IOASID ) { +pr_err ("Failed ASID allocation %lu\n" , active_allocator->flags); goto exit_free; } + +if (active_allocator->flags &IOASID_ALLOCATOR_CUSTOM ) { +/* Custom allocator needs framework to store and track allocation results */ + min = max = id; + +if (xa_alloc (&active_allocator->xa, &id, data,XA_LIMIT (min, max),GFP_KERNEL )) { +pr_err ("Failed to alloc ioasid from %d to %d\n" , min, max); + active_allocator->ops->free (id,NULL ); + goto exit_free; + } + } data->id = id; +spin_unlock (&ioasid_allocator_lock);return id; exit_free: +spin_unlock (&ioasid_allocator_lock);kfree (data);return INVALID_IOASID ; }
@@ -97,9 +365,22 @@ void ioasid_free(ioasid_t ioasid) {struct ioasid_data *ioasid_data; - ioasid_data =xa_erase (&ioasid_xa, ioasid); +spin_lock (&ioasid_allocator_lock); + ioasid_data =xa_load (&active_allocator->xa, ioasid); +if (!ioasid_data) { +pr_err ("Trying to free unknown IOASID %u\n" , ioasid); + goto exit_unlock; + } -kfree_rcu (ioasid_data, rcu); + active_allocator->ops->free (ioasid, active_allocator->ops->pdata); +/* Custom allocator needs additional steps to free the xa element */ +if (active_allocator->flags &IOASID_ALLOCATOR_CUSTOM ) { + ioasid_data =xa_erase (&active_allocator->xa, ioasid); +kfree_rcu (ioasid_data, rcu); + } + +exit_unlock: +spin_unlock (&ioasid_allocator_lock); }EXPORT_SYMBOL_GPL (ioasid_free);
@@ -124,7 +405,7 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,struct ioasid_data *ioasid_data;rcu_read_lock (); - ioasid_data =xa_load (&ioasid_xa, ioasid); + ioasid_data =xa_load (&active_allocator->xa, ioasid);if (!ioasid_data) { priv =ERR_PTR (-ENOENT ); goto unlock;
include/linux/ioasid.h +28 -0
@@ -6,11 +6,28 @@#define INVALID_IOASID ((ioasid_t)-1)typedef unsigned int ioasid_t ; +typedef ioasid_t (*ioasid_alloc_fn_t )(ioasid_t min,ioasid_t max,void *data); +typedef void (*ioasid_free_fn_t )(ioasid_t ioasid,void *data);struct ioasid_set {int dummy; }; +/** +* struct ioasid_allocator_ops - IOASID allocator helper functions and data +* +* @alloc: helper function to allocate IOASID +* @free: helper function to free IOASID +* @list: for tracking ops that share helper functions but not data +* @pdata: data belong to the allocator, provided when calling alloc() +*/ +struct ioasid_allocator_ops { +ioasid_alloc_fn_t alloc; +ioasid_free_fn_t free; +struct list_head list; +void *pdata; +}; +#define DECLARE_IOASID_SET (name) struct ioasid_set name = { 0 }#if IS_ENABLED (CONFIG_IOASID )
@@ -19,6 +36,8 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,void ioasid_free (ioasid_t ioasid);void *ioasid_find (struct ioasid_set *set,ioasid_t ioasid,bool (*getter)(void *)); +int ioasid_register_allocator (struct ioasid_allocator_ops *allocator); +void ioasid_unregister_allocator (struct ioasid_allocator_ops *allocator);int ioasid_set_data (ioasid_t ioasid,void *data);#else /* !CONFIG_IOASID */
@@ -38,6 +57,15 @@ static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,return NULL ; } +static inline int ioasid_register_allocator (struct ioasid_allocator_ops *allocator) +{ +return -ENODEV ; +} + +static inline void ioasid_unregister_allocator (struct ioasid_allocator_ops *allocator) +{ +} +static inline int ioasid_set_data (ioasid_t ioasid,void *data) {return -ENODEV ;
Signature · 1 line
2.7.4
Re: [PATCH 3/4] iommu/ioasid: Add custom allocators
Quoted 8 lines · Jacob Pan
+/*
+ * structioasid_allocator_data- Internal data structure to hold information
+ * about an allocator. There are two types of allocators:
+ *
+ * - Default allocator always has its own XArray to track the IOASIDs allocated.
+ * - Custom allocators may share allocation helpers with different private data.
+ * Custom allocators share the same helper functions also share the same
+ * XArray.
"that share the same helper"
Quoted 21 lines
+ * Rules:
+ * 1. Default allocator is always available, not dynamically registered. This is
+ * to prevent race conditions with early boot code that want to register
+ * custom allocators or allocate IOASIDs.
+ * 2. Custom allocators take precedence over the default allocator.
+ * 3. When all custom allocators sharing the same helper functions are
+ * unregistered (e.g. due to hotplug), all outstanding IOASIDs must be
+ * freed.
+ * 4. When switching between custom allocators sharing the same helper
+ * functions, outstanding IOASIDs are preserved.
+ * 5. When switching between custom allocator and default allocator, all IOASIDs
+ * must be freed to ensure unadulterated space for the new allocator.
+ *
+ * @ops: allocator helper functions and its data
+ * @list: registered custom allocators
+ * @slist: allocators share the same ops but different data
+ * @flags: attributes of the allocator
+ * @xa xarray holds the IOASID space
+ * @users number of allocators sharing the same ops and XArray
+ */drivers/iommu/ioasid.c +1
-0+struct ioasid_allocator_data {
drivers/iommu/ioasid.c struct ioasid_data { · line 58
Quoted 10 lines
drivers/iommu/ioasid.c +10
-0+struct ioasid_allocator_ops *ops; +struct list_head list; +struct list_head slist; +#define IOASID_ALLOCATOR_CUSTOM BIT(0)/* Needs framework to track results */ +unsigned long flags; +struct xarray xa; +refcount_t users; +}; + +static DEFINE_SPINLOCK (ioasid_allocator_lock);
Thanks for making this a spinlock! I hit that sleep-in-atomic problem while updating iommu-sva to the new MMU notifier API, which doesn't allow sleeping in the free() callback.
I don't like having to use GFP_ATOMIC everywhere as a result, but can't see a better way. Maybe we can improve that later.
[...]
Quoted 22 lines
drivers/iommu/ioasid.c +22
-0+/** +* ioasid_unregister_allocator - Remove a custom IOASID allocator ops +* @ops: the custom allocator to be removed +* +* Remove an allocator from the list, activate the next allocator in +* the order it was registered. Or revert to default allocator if all +* custom allocators are unregistered without outstanding IOASIDs. +*/ +void ioasid_unregister_allocator (struct ioasid_allocator_ops *ops) +{ +struct ioasid_allocator_data *pallocator; +struct ioasid_allocator_ops *sops; + +spin_lock (&ioasid_allocator_lock); +if (list_empty (&allocators_list)) { +pr_warn ("No custom IOASID allocators active!\n" ); + goto exit_unlock; + } + +list_for_each_entry (pallocator, &allocators_list, list) { +if (!use_same_ops (pallocator->ops, ops)) +continue ;
drivers/iommu/ioasid.c struct ioasid_data { · line 238
Quoted 10 lines
drivers/iommu/ioasid.c +10
-0+ +if (refcount_read (&pallocator->users) ==1 ) { +/* No shared helper functions */ +list_del (&pallocator->list); +/* +* All IOASIDs should have been freed before +* the last allocator that shares the same ops +* is unregistered. +*/ +WARN_ON (!xa_empty (&pallocator->xa));
The function doc seems to say that we revert to the default allocator only if there wasn't any outstanding IOASID, which isn't what this does. To follow the doc, we'd need to return here instead of continuing. The best solution would be to return with an error, but since we don't propagate errors I think leaking stuff is preferable to leaving the allocator registered, since the caller might free the ops when this function return. So I would keep the code like that but change the function's comment. What do you think is best?
drivers/iommu/ioasid.c struct ioasid_data { · line 242
Quoted 4 lines
drivers/iommu/ioasid.c +4
-0+kfree (pallocator); +if (list_empty (&allocators_list)) { +pr_info ("No custom IOASID allocators, switch to default.\n" ); + active_allocator = &default_allocator;
I'm concerned about the active_allocator variable, because ioasid_find() accesses it without holding ioasid_allocator_lock. It is holding the RCU read lock, so I think we need to free pallocator after a RCU grace period (using kfree_rcu)? I think we also need to update active_allocator with rcu_assign_pointer() and dereference it with rcu_dereference()
Quoted 5 lines
drivers/iommu/ioasid.c +5
-0+ }else if (pallocator == active_allocator) { + active_allocator = list_first_entry(&allocators_list, struct ioasid_allocator_data, list); +pr_info ("IOASID allocator changed" ); + } +break ;
drivers/iommu/ioasid.c struct ioasid_data { · line 257
Quoted 10 lines
drivers/iommu/ioasid.c +10
-0+ } +/* +* Find the matching shared ops to delete, +* but keep outstanding IOASIDs +*/ +list_for_each_entry (sops, &pallocator->slist, list) { +if (sops == ops) { +list_del (&ops->list); +if (refcount_dec_and_test (&pallocator->users)) +pr_err ("no shared ops\n" );
That's not possible, right, since dec_and_test only returns true if pallocator->users was 1, which we already checked against? I find pallocator->users a bit redundant since you can use list_is_empty() or list_is_singular() on pallocator->slist
[...]
Quoted 23 lines
ioasid_tioasid_alloc(structioasid_set*set,ioasid_tmin,ioasid_tmax,drivers/iommu/ioasid.c +9
-3void *private) { -ioasid_t id;struct ioasid_data *data; +void *adata; +ioasid_t id; data =kzalloc (sizeof (*data),GFP_KERNEL );if (!data)@@ -76,14 +324,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; -if (xa_alloc (&ioasid_xa, &id, data,XA_LIMIT (min, max),GFP_KERNEL )) { -pr_err ("Failed to alloc ioasid from %d to %d\n" , min, max); +/* +* Custom allocator needs allocator data to perform platform specific +* operations. +*/ +spin_lock (&ioasid_allocator_lock); + adata = active_allocator->flags &IOASID_ALLOCATOR_CUSTOM ? active_allocator->ops->pdata : data; + id = active_allocator->ops->alloc (min, max, adata);
drivers/iommu/ioasid.c ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 343
Quoted 10 lines
drivers/iommu/ioasid.c +8
-0+if (id ==INVALID_IOASID ) { +pr_err ("Failed ASID allocation %lu\n" , active_allocator->flags); goto exit_free; } + +if (active_allocator->flags &IOASID_ALLOCATOR_CUSTOM ) { +/* Custom allocator needs framework to store and track allocation results */ + min = max = id; + +if (xa_alloc (&active_allocator->xa, &id, data,XA_LIMIT (min, max),GFP_KERNEL )) {
Or just XA_LIMIT(id, id), and merge the two ifs?
You do need GFP_ATOMIC here.
Review ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 344
Quoted 1 line
(quoted diff) +1
-0+ pr_err("Failed to alloc ioasid from %d to %d\n", min, max);
Maybe just "Failed to alloc ioasid %d\n" then
drivers/iommu/ioasid.c ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 345
Quoted 1 line
drivers/iommu/ioasid.c +1
-0+ active_allocator->ops->free (id,NULL );
Why doesn't this call need to pass active_allocator->ops->pdata like the one in ioasid_free()?
Thanks,
Jean
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 3/4] iommu/ioasid: Add custom allocators
Quoted 103 lines · On Fri, 20 Sep 2019 18:35:58 +0200 Jean-Philippe Brucker…
Quoted 4 lines · Jacob Pan
+/*
+ * structioasid_allocator_data- Internal data structure to hold information
+ * about an allocator. There are two types of allocators:Quoted 8 lines
+ *
+ * - Default allocator always has its own XArray to track the IOASIDs allocated.
+ * - Custom allocators may share allocation helpers with different private data.
+ * Custom allocators share the same helper functions also share
the same
+ * XArray."that share the same helper"
Quoted 34 lines
+ * Rules:
+ * 1. Default allocator is always available, not dynamically registered. This is
+ * to prevent race conditions with early boot code that want to
register
+ * custom allocators or allocate IOASIDs.
+ * 2. Custom allocators take precedence over the default allocator.
+ * 3. When all custom allocators sharing the same helper functions are
+ * unregistered (e.g. due to hotplug), all outstanding IOASIDs
must be
+ * freed.
+ * 4. When switching between custom allocators sharing the same helper
+ * functions, outstanding IOASIDs are preserved.
+ * 5. When switching between custom allocator and default allocator, all IOASIDs
+ * must be freed to ensure unadulterated space for the new
allocator.
+ *
+ * @ops: allocator helper functions and its data
+ * @list: registered custom allocators
+ * @slist: allocators share the same ops but different data
+ * @flags: attributes of the allocator
+ * @xa xarray holds the IOASID space
+ * @users number of allocators sharing the same ops and
XArray
+ */drivers/iommu/ioasid.c +5
-0+struct ioasid_allocator_data { +struct ioasid_allocator_ops *ops; +struct list_head list; +struct list_head slist; +#define IOASID_ALLOCATOR_CUSTOM BIT(0)/* Needs framework to track results */
drivers/iommu/ioasid.c struct ioasid_data { · line 58
Quoted 6 lines
drivers/iommu/ioasid.c +6
-0+unsigned long flags; +struct xarray xa; +refcount_t users; +}; + +static DEFINE_SPINLOCK (ioasid_allocator_lock);Thanks for making this a spinlock! I hit that sleep-in-atomic problem while updating iommu-sva to the new MMU notifier API, which doesn't allow sleeping in the
free()callback.I don't like having to use
GFP_ATOMICeverywhere as a result, but can't see a better way. Maybe we can improve that later.[...]
Quoted 34 lines
+/**
+ *ioasid_unregister_allocator- Remove a custom IOASID allocator ops
+ * @ops: the custom allocator to be removed
+ *
+ * Remove an allocator from the list, activate the next allocator in
+ * the order it was registered. Or revert to default allocator if all
+ * custom allocators are unregistered without outstanding IOASIDs.
+ */drivers/iommu/ioasid.c +20
-0+void ioasid_unregister_allocator (struct ioasid_allocator_ops *ops) +{ +struct ioasid_allocator_data *pallocator; +struct ioasid_allocator_ops *sops; + +spin_lock (&ioasid_allocator_lock); +if (list_empty (&allocators_list)) { +pr_warn ("No custom IOASID allocators active!\n" ); + goto exit_unlock; + } + +list_for_each_entry (pallocator, &allocators_list, list) { +if (!use_same_ops (pallocator->ops, ops)) +continue ; + +if (refcount_read (&pallocator->users) ==1 ) { +/* No shared helper functions */ +list_del (&pallocator->list); +/* +* All IOASIDs should have been freed before
(quoted diff) +1
-0+ * the last allocator that shares the sameops
drivers/iommu/ioasid.c struct ioasid_data { · line 238
Quoted 3 lines
drivers/iommu/ioasid.c +3
-0+* is unregistered. +*/ +WARN_ON (!xa_empty (&pallocator->xa));
Quoted 9 lines
The function doc seems to say that we revert to the default allocator only if there wasn't any outstanding IOASID, which isn't what this does. To follow the doc, we'd need to return here instead of continuing. The best solution would be to return with an error, but since we don't propagate errors I think leaking stuff is preferable to leaving the allocator registered, since the caller might free the ops when this function return. So I would keep the code like that but change the function's comment. What do you think is best?
I agree, unregister allocator should not fail. I will change the comments stating that if allocator is unregistered prior to freeing all outstanding IOASIDs, these IOASIDs will be orphaned and lost.
Quoted 5 lines
Quoted 5 lines
drivers/iommu/ioasid.c +3
-0+kfree (pallocator); +if (list_empty (&allocators_list)) { + pr_info("No custom IOASIDallocators, switch to default.\n");
(quoted diff) +1
-0+ active_allocator =
Quoted 9 lines
Quoted 1 line
&default_allocator;I'm concerned about the
active_allocatorvariable, becauseioasid_find()accesses it without holdingioasid_allocator_lock. It is holding the RCU read lock, so I think we need to free pallocator after a RCU grace period (usingkfree_rcu)? I think we also need to updateactive_allocatorwithrcu_assign_pointer()and dereference it withrcu_dereference()
right, will do. ioasid_find is on the fast path, so we try not to use spinlock.
Quoted 19 lines
Quoted 19 lines
(quoted diff) +1
-0+ } else if (pallocator == active_allocator){
(quoted diff) +1
-0+ active_allocator =
list_first_entry(&allocators_list, struct ioasid_allocator_data, list);(quoted diff) +1
-0+ pr_info("IOASID allocatorchanged");
drivers/iommu/ioasid.c +8
-0+ } +break ; + } +/* +* Find the matching shared ops to delete, +* but keep outstanding IOASIDs +*/ + list_for_each_entry(sops, &pallocator->slist,list) {
drivers/iommu/ioasid.c +3
-0+if (sops == ops) { +list_del (&ops->list); + if
Quoted 9 lines
Quoted 3 lines
(
refcount_dec_and_test(&pallocator->users))(quoted diff) +1
-0+ pr_err("no sharedops\n");
That's not possible, right, since
dec_and_testonly returns true ifpallocator->userswas 1, which we already checked against? I findpallocator->usersa bit redundant since you can uselist_is_empty()orlist_is_singular()onpallocator->slist
you are right, no need for the refcount, just check the status of the shared op list.
Quoted 34 lines
[...]
Quoted 33 lines
ioasid_tioasid_alloc(structioasid_set*set,ioasid_tmin,ioasid_tmax, void *private)
{drivers/iommu/ioasid.c +4
-3-ioasid_t id;struct ioasid_data *data; +void *adata; +ioasid_t id; data =kzalloc (sizeof (*data),GFP_KERNEL );if (!data)@@ -76,14 +324,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; -if (xa_alloc (&ioasid_xa, &id, data,XA_LIMIT (min, max), GFP_KERNEL)) { -pr_err ("Failed to alloc ioasid from %d to %d\n" , min, max); +/* +* Custom allocator needs allocator data to perform platform specific
drivers/iommu/ioasid.c +4
-0+* operations. +*/ +spin_lock (&ioasid_allocator_lock); + adata = active_allocator->flags &
IOASID_ALLOCATOR_CUSTOM?active_allocator->ops->pdata: data;drivers/iommu/ioasid.c +3
-0+ id = active_allocator->ops->alloc (min, max, adata); +if (id ==INVALID_IOASID ) { + pr_err("Failed ASID allocation %lu\n",
active_allocator->flags); gotoexit_free;drivers/iommu/ioasid.c +2
-0} + +if (active_allocator->flags &IOASID_ALLOCATOR_CUSTOM ) {
Quoted 9 lines
Quoted 6 lines
(quoted diff) +1
-0+ /* Custom allocator needs framework to store andtrack allocation results */
drivers/iommu/ioasid.c +3
-0+ min = max = id; + + if (xa_alloc(&active_allocator->xa, &id, data,
XA_LIMIT(min, max),GFP_KERNEL)) {Or just
XA_LIMIT(id, id), and merge the two ifs?
much better, thanks for the suggestions.
Quoted 2 lines
You do need
GFP_ATOMIChere.
right, will change.
Quoted 5 lines
Quoted 2 lines
(quoted diff) +1
-0+ pr_err("Failed to alloc ioasid from %d to%d\n", min, max);
Maybe just "Failed to alloc ioasid %d\n" then
agreed. just failed on a specific id, not range.
Quoted 5 lines
drivers/iommu/ioasid.c ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 345
Quoted 1 line
drivers/iommu/ioasid.c +1
-0+ active_allocator->ops->free (id,NULL );Why doesn't this call need to pass
active_allocator->ops->pdatalike the one inioasid_free()?
Good catch, this call also need pdata.
Quoted 2 lines
Thanks,
Jean
PATCH 4/4 iommu: Introduce guest PASID bind function
In reply to Jacob PanGuest shared virtual address (SVA) may require host to shadow guest PASID tables. Guest PASID can also be allocated from the host via enlightened interfaces. In this case, guest needs to bind the guest mm, i.e. cr3 in guest physical address to the actual PASID table in the host IOMMU. Nesting will be turned on such that guest virtual address can go through a two level translation:
- 1st level translates GVA to GPA
- 2nd level translates GPA to HPA
This patch introduces APIs to bind guest PASID data to the assigned device entry in the physical IOMMU. See the diagram below for usage explaination.
.-------------. .---------------------------.
| vIOMMU | | Guest process mm, FL only |
| | '---------------------------' .----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | GP
'-------------'
Guest
------| Shadow |----------------------- GP->HP* ---------
v v |
Host v
.-------------. .----------------------.
| pIOMMU | | Bind FL for GVA-GPA |
| | '----------------------'
.----------------/ |
| PASID Entry | V (Nested xlate)
'----------------\.---------------------.
| | |Set SL to GPA-HPA |
| | '---------------------'
'-------------'Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables
- GP = Guest PASID
- HP = Host PASID
* Conversion needed if non-identity GP-HP mapping option is chosen.
--- drivers/iommu/iommu.c | 20 ++++++++++++++++ include/linux/iommu.h | 22 +++++++++++++++++ include/uapi/linux/iommu.h | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 101 insertions(+)
drivers/iommu/iommu.c +20 -0
@@ -1620,6 +1620,26 @@ int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev, }EXPORT_SYMBOL_GPL (iommu_cache_invalidate); +int iommu_sva_bind_gpasid (struct iommu_domain *domain, +struct device *dev,struct iommu_gpasid_bind_data *data) +{ +if (unlikely (!domain->ops->sva_bind_gpasid)) +return -ENODEV ; + +return domain->ops->sva_bind_gpasid (domain, dev, data); +} +EXPORT_SYMBOL_GPL (iommu_sva_bind_gpasid); + +int iommu_sva_unbind_gpasid (struct iommu_domain *domain,struct device *dev, +ioasid_t pasid) +{ +if (unlikely (!domain->ops->sva_unbind_gpasid)) +return -ENODEV ; + +return domain->ops->sva_unbind_gpasid (dev, pasid); +} +EXPORT_SYMBOL_GPL (iommu_sva_unbind_gpasid); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {
include/linux/iommu.h +22 -0
@@ -13,6 +13,7 @@#include <linux/errno.h> #include <linux/err.h> #include <linux/of.h> +#include <linux/ioasid.h> #include <uapi/linux/iommu.h> #define IOMMU_READ (1 << 0)
@@ -230,6 +231,8 @@ struct iommu_sva_ops {* @page_response: handle page request response * @cache_invalidate: invalidate translation caches * @pgsize_bitmap: bitmap of all possible supported page sizes +* @sva_bind_gpasid: bind guest pasid and mm +* @sva_unbind_gpasid: unbind guest pasid and mm */ struct iommu_ops {bool (*capable)(enum iommu_cap );
@@ -294,6 +297,10 @@ struct iommu_ops { struct iommu_page_response *msg);int (*cache_invalidate)(struct iommu_domain *domain,struct device *dev,struct iommu_cache_invalidate_info *inv_info); +int (*sva_bind_gpasid)(struct iommu_domain *domain, +struct device *dev,struct iommu_gpasid_bind_data *data); + +int (*sva_unbind_gpasid)(struct device *dev,int pasid);unsigned long pgsize_bitmap; };
@@ -401,6 +408,10 @@ extern void iommu_detach_device(struct iommu_domain *domain,extern int iommu_cache_invalidate (struct iommu_domain *domain,struct device *dev,struct iommu_cache_invalidate_info *inv_info); +extern int iommu_sva_bind_gpasid (struct iommu_domain *domain, +struct device *dev,struct iommu_gpasid_bind_data *data); +extern int iommu_sva_unbind_gpasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid);extern struct iommu_domain *iommu_get_domain_for_dev (struct device *dev);extern struct iommu_domain *iommu_get_dma_domain (struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
@@ -950,6 +961,17 @@ iommu_cache_invalidate(struct iommu_domain *domain, {return -ENODEV ; } +static inline int iommu_sva_bind_gpasid (struct iommu_domain *domain, +struct device *dev,struct iommu_gpasid_bind_data *data) +{ +return -ENODEV ; +} + +static inline int iommu_sva_unbind_gpasid (struct iommu_domain *domain, +struct device *dev,int pasid) +{ +return -ENODEV ; +}#endif /* CONFIG_IOMMU_API */
include/uapi/linux/iommu.h +59 -0
63 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH 4/4] iommu: Introduce guest PASID bind function
Quoted 11 lines · Jacob Pan
Guest shared virtual address (SVA) may require host to shadow guest PASID tables. Guest PASID can also be allocated from the host via enlightened interfaces. In this case, guest needs to bind the guest mm, i.e. cr3 in guest physical address to the actual PASID table in the host IOMMU. Nesting will be turned on such that guest virtual address can go through a two level translation:
- 1st level translates GVA to GPA
- 2nd level translates GPA to HPA
This patch introduces APIs to bind guest PASID data to the assigned device entry in the physical IOMMU. See the diagram below for usage explaination.
explanation
Otherwise Looks fine to me. I was wondering if we would be able to reuse the API for Arm SMMUv2, which allows nesting translation, but without PASID - there is a single address space per device, with two stages of translation. I think it would work, although it would look better with something like "PGD" instead of "PASID" in the API names (e.g. iommu_sva_bind_gpgd) since that case wouldn't use PASID at all. But I don't want to quibble over names, so
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu