vfio: expose virtual Shared Virtual Addressing to VMs
Subsystems (2)
cover vfio: expose virtual Shared Virtual Addressing to VMs
From: Liu Yi L <[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
This VFIO series is intended to expose SVA usage to VMs. i.e. Sharing guest application address space with passthru devices. This is called vSVA in this series. The whole vSVA enabling requires QEMU/VFIO/IOMMU changes. For IOMMU and QEMU changes, they are in separate series (listed in the "Related series").
The high-level architecture for SVA virtualization is as below, the key design of vSVA support is to utilize the dual-stage IOMMU translation ( also known as IOMMU nesting translation) capability in host IOMMU.
.-------------. .---------------------------.
| vIOMMU | | Guest process CR3, FL only|
| | '---------------------------' .----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest
------| Shadow |--------------------------|--------
v v v
Host
.-------------. .----------------------.
| pIOMMU | | Bind FL for GVA-GPA |
| | '----------------------'
.----------------/ |
| PASID Entry | V (Nested xlate)
'----------------\.------------------------------.
| | |SL for GPA-HPA, default domain|
| | '------------------------------'
'-------------'Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables
There are roughly four parts in this patchset which are corresponding to the basic vSVA support for PCI device assignment
1. vfio support for PASID allocation and free for VMs
2. vfio support for guest page table binding request from VMs
3. vfio support for IOMMU cache invalidation from VMs
4. vfio support for vSVA usage on IOMMU-backed mdevs
The complete vSVA kernel upstream patches are divided into three phases:
1. Common APIs and PCI device direct assignment
2. IOMMU-backed Mediated Device assignment
3. Page Request Services (PRS) support
This patchset is aiming for the phase 1 and phase 2, and based on Jacob's below series.
[PATCH V10 00/11] Nested Shared Virtual Address (SVA) VT-d support:
https://lkml.org/lkml/2020/3/20/1172
Complete set for current vSVA can be found in below branch.
https://github.com/luxis1999/linux-vsva.git: vsva-linux-5.6-rc6
The corresponding QEMU patch series is as below, complete QEMU set can be found in below branch.
[PATCH v1 00/22] intel_iommu: expose Shared Virtual Addressing to VMs complete QEMU set can be found in below link:
https://github.com/luxis1999/qemu.git: sva_vtd_v10_v1
Regards,
Yi Liu
Changelog:
- RFC v1 -> Patch v1:
a) Address comments to the PASID request(alloc/free) path
b) Report PASID alloc/free availabitiy to user-space
c) Add a vfio_iommu_type1 parameter to support pasid quota tuning
d) Adjusted to latest ioasid code implementation. e.g. remove the code for tracking the allocated PASIDs as latest ioasid code will track it, VFIO could use ioasid_free_set() to free all PASIDs.
- RFC v2 -> v3:
a) Refine the whole patchset to fit the roughly parts in this series
b) Adds complete vfio PASID management framework. e.g. pasid alloc, free, reclaim in VM crash/down and per-VM PASID quota to prevent PASID abuse.
c) Adds IOMMU uAPI version check and page table format check to ensure version compatibility and hardware compatibility.
d) Adds vSVA vfio support for IOMMU-backed mdevs.
- RFC v1 -> v2:
Dropped vfio: VFIO_IOMMU_ATTACH/DETACH_PASID_TABLE.
Liu Yi L (8):
vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free) vfio/type1: Add vfio_iommu_type1 parameter for quota tuning vfio/type1: Report PASID alloc/free support to userspace
vfio: Check nesting iommu uAPI version vfio/type1: Report 1st-level/stage-1 format to userspace vfio/type1: Bind guest page tables to host vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE vfio/type1: Add vSVA support for IOMMU-backed mdevs
drivers/vfio/vfio.c | 136 +++++++++++++ drivers/vfio/vfio_iommu_type1.c | 419 ++++++++++++++++++++++++++++++++++++++++ include/linux/vfio.h | 21 ++ include/uapi/linux/vfio.h | 127 ++++++++++++ 4 files changed, 703 insertions(+)
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 1/8 vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
This patch adds VFIO_IOMMU_PASID_REQUEST ioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and vfio_iommu structure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While for vfio_iommu structure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structure vfio_mm. A vfio_mm is created when the first vfio container is opened by a VM. On the reverse order, vfio_mm is free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/
--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)
drivers/vfio/vfio.c +130 -0
@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"
@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;
@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} + +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true); +mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ +kref_put_mutex (&vmm->kref, vfio_mm_release, &vfio.vfio_mm_lock); +} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm); +return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)
@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);
drivers/vfio/vfio_iommu_type1.c +104 -0
140 lines not on this page · raw message
include/linux/vfio.h +20 -0
@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */
include/uapi/linux/vfio.h +41 -0
47 lines not on this page · raw message
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
kbuild test robot
build botRe: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Thank you for the patch! Perhaps something to improve:
Hi Yi,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on vfio/next]
[also build test WARNING on v5.6-rc6 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Liu-Yi-L/vfio-expose-virtual-Shared-Virtual-Addressing-to-VMs/20200322-213259 base: https://github.com/awilliam/linux-vfio.git next config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag
All warnings (new ones prefixed by >>):
drivers/vfio/vfio.c: In function 'vfio_create_mm':
drivers/vfio/vfio.c:2149:8: error: implicit declaration of function 'ioasid_alloc_set'; did you mean 'ioasid_alloc'? [-Werror=implicit-function-declaration]
2149 | ret = ioasid_alloc_set((struct ioasid_set *) mm,
| ^~~~~~~~~~~~~~~~
| ioasid_alloc
>> drivers/vfio/vfio.c:2158:13: warning: assignment to 'long long unsigned int' from 'struct mm_struct *' makes integer from pointer without a cast [-Wint-conversion]
2158 | token->val = mm;
| ^
drivers/vfio/vfio.c: In function 'vfio_mm_unlock_and_free':
drivers/vfio/vfio.c:2170:2: error: implicit declaration of function 'ioasid_free_set'; did you mean 'ioasid_free'? [-Werror=implicit-function-declaration]
2170 | ioasid_free_set(vmm->ioasid_sid, true);
| ^~~~~~~~~~~~~~~
| ioasid_free
drivers/vfio/vfio.c: In function 'vfio_mm_pasid_alloc':
drivers/vfio/vfio.c:2227:26: warning: passing argument 1 of 'ioasid_alloc' makes pointer from integer without a cast [-Wint-conversion]
2227 | pasid = ioasid_alloc(vmm->ioasid_sid, min, max, NULL);
| ~~~^~~~~~~~~~~~
| |
| int
In file included from include/linux/iommu.h:16,
from drivers/vfio/vfio.c:20:
include/linux/ioasid.h:45:56: note: expected 'struct ioasid_set *' but argument is of type 'int'
45 | static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
| ~~~~~~~~~~~~~~~~~~~^~~
drivers/vfio/vfio.c: In function 'vfio_mm_pasid_free':
drivers/vfio/vfio.c:2246:25: warning: passing argument 1 of 'ioasid_find' makes pointer from integer without a cast [-Wint-conversion]
2246 | pdata = ioasid_find(vmm->ioasid_sid, pasid, NULL);
| ~~~^~~~~~~~~~~~
| |
| int In file included from include/linux/iommu.h:16,
from drivers/vfio/vfio.c:20:
include/linux/ioasid.h:55:52: note: expected 'struct ioasid_set *' but argument is of type 'int'
55 | static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
| ~~~~~~~~~~~~~~~~~~~^~~
cc1: some warnings being treated as errors
vim +2158 drivers/vfio/vfio.c
2133
2134 /**
2135 * VFIO_MM objects - create, release, get, put, search
2136 * Caller of the function should have held vfio.vfio_mm_lock.
2137 */
2138 static struct vfio_mm *vfio_create_mm(struct mm_struct *mm)
2139 {
2140 struct vfio_mm *vmm;
2141 struct vfio_mm_token *token;
2142 int ret = 0;
2143
2144 vmm = kzalloc(sizeof(*vmm), GFP_KERNEL);
2145 if (!vmm)
2146 return ERR_PTR(-ENOMEM);
2147
2148 /* Per mm IOASID set used for quota control and group operations */
2149 ret = ioasid_alloc_set((struct ioasid_set *) mm,
2150 VFIO_DEFAULT_PASID_QUOTA, &vmm->ioasid_sid);
2151 if (ret) {
2152 kfree(vmm);
2153 return ERR_PTR(ret);
2154 }
2155
2156 kref_init(&vmm->kref);
2157 token = &vmm->token;
> 2158 token->val = mm;
2159 vmm->pasid_quota = VFIO_DEFAULT_PASID_QUOTA;
2160 mutex_init(&vmm->pasid_lock);
2161
2162 list_add(&vmm->vfio_next, &vfio.vfio_mm_list);
2163
2164 return vmm;
2165 }
2166
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Liu, Yi LQuoted 9 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity
are->is
Quoted 6 lines
DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the
"address space" -> "address spaces"
"binding the" -> "binding each"
Quoted 12 lines
virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 3 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown
PASID allocation/free request from the virtual IOMMU. Additionally, such
"Additionally, because such"
Quoted 1 line
requests are intended to be invoked by QEMU or other applications which
simplify to "intended to be invoked from userspace"
Quoted 6 lines
are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is
devices -> device
Quoted 59 lines
possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +1
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamsondrivers/vfio/vfio.c +2
-0@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, · line 2136
Quoted 5 lines
drivers/vfio/vfio.c +1
-0@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev,enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier );/** +* VFIO_MM objects - create, release, get, put, search
why capitalizing vfio_mm?
Quoted 27 lines
drivers/vfio/vfio.c +25
-0+* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm- >ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock);
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2170
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+ +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} + +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true);
do we need hold pasid lock here, since it attempts to destroy a set which might be referenced by vfio_mm_pasid_free? or is there guarantee that such race won't happen?
Quoted 36 lines
drivers/vfio/vfio.c +16
-0+mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ + kref_put_mutex(&vmm->kref, vfio_mm_release,&vfio.vfio_mm_lock);
drivers/vfio/vfio.c +19
-0+} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm);
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2215
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+ goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm);
I assume this has been discussed before, but from readability p.o.v it might be good to add a comment for this function to explain how the recording of mm in vfio_mm can be correctly removed when the mm is being destroyed, since we don't hold a reference of mm here.
Quoted 81 lines
drivers/vfio/vfio.c +46
-0+return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);drivers/vfio/vfio_iommu_type1.c +2
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };struct vfio_domain {@@ -2018,6 +2019,7 @@ static void vfio_iommu_type1_detach_group(void *iommu_data, static void *vfio_iommu_type1_open(unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );drivers/vfio/vfio_iommu_type1.c +0
-0if (!iommu)
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned · line 2051
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0@@ -2043,6 +2045,10 @@ static void *vfio_iommu_type1_open(unsigned long arg) iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );
I assume error should be returned when pr_err is used...
Quoted 15 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu;}
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void *iommu_data) }vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu);}
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct · line 2184
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +1
-0@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(structvfio_iommu *iommu, return ret; } +static bool vfio_iommu_type1_pasid_req_valid (u32 flags)
I don't think you need prefix "vfio_iommu_type1" for every new function here, especially for leaf internal function as this one.
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} +
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2199
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+static int vfio_iommu_type1_pasid_alloc (struct vfio_iommu *iommu, +int min, +int max) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;
why -EFAULT?
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ goto out_unlock; + } +if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +}
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2219
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +static int vfio_iommu_type1_pasid_free (struct vfio_iommu *iommu, +unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;
ditto
Quoted 29 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ goto out_unlock; + } + +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg){
drivers/vfio/vfio_iommu_type1.c +9
-0@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, return copy_to_user((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz))
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void · line 2355
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags)) +return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;
why copying in two steps instead of copying them together?
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +11
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof(
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2373
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+struct vfio_iommu_type1_pasid_request, + alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result)); + }else { +pr_debug ("%s: PASID alloc failed\n" , __func__); + ret = -EFAULT ;
no, this branch is not for copy_to_user error. it is about pasid alloc failure. you should handle both.
Quoted 63 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+ } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + }drivers/vfio/vfio_iommu_type1.c +0
-0}return -ENOTTY ;include/linux/vfio.h +20
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */
include/uapi/linux/vfio.h +11
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 817
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+struct vfio_iommu_type1_pasid_request { + __u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0) +#define VFIO_IOMMU_PASID_FREE (1 << 1) + __u32 flags; + union { + struct { + __u32 min; + __u32 max; + __u32 result;
result->pasid?
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 819
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+ } alloc_pasid; +__u32 free_pasid;
what about putting a common pasid field after flags?
Quoted 24 lines
include/uapi/linux/vfio.h +18
-0+ }; +}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC | \ + VFIO_IOMMU_PASID_FREE) + +/** +* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it +* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
--
2.7.4
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 8 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:32 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 3 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PM
Quoted 8 lines
Quoted 6 lines
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity
are->is
thanks.
Quoted 10 lines
Quoted 6 lines
DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the
"address space" -> "address spaces"
"binding the" -> "binding each"
will correct both.
Quoted 16 lines
Quoted 12 lines
virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 3 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown
PASID allocation/free request from the virtual IOMMU. Additionally, such
Quoted 5 lines
"Additionally, because such"
Quoted 1 line
requests are intended to be invoked by QEMU or other applications which
simplify to "intended to be invoked from userspace"
got it.
Quoted 8 lines
Quoted 6 lines
are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is
devices -> device
got it.
Quoted 58 lines
Quoted 58 lines
possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +1
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamsondrivers/vfio/vfio.c +2
-0@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;
Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, · line 2136
Quoted 6 lines
drivers/vfio/vfio.c +1
-0@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev,enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier );/** +* VFIO_MM objects - create, release, get, put, search why capitalizing
vfio_mm?
oops, it's not intended, will fix it.
Quoted 33 lines
Quoted 33 lines
drivers/vfio/vfio.c +31
-0+* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm- >ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} +
Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2170
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true);do we need hold pasid lock here, since it attempts to destroy a set which might be referenced by
vfio_mm_pasid_free? or is there guarantee that such race won't happen?
Emmm, if considering the race between ioasid_free_set and vfio_mm_pasid_free, I guess ioasid core should sequence the two operations with its internal lock. right?
Quoted 44 lines
Quoted 44 lines
drivers/vfio/vfio.c +16
-0+mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ + kref_put_mutex(&vmm->kref, vfio_mm_release,&vfio.vfio_mm_lock);
drivers/vfio/vfio.c +27
-0+} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out:
Quoted 8 lines
drivers/vfio/vfio.c
Quoted 2 lines
drivers/vfio/vfio.c +2
-0+mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm);I assume this has been discussed before, but from readability p.o.v it might be good to add a comment for this function to explain how the recording of mm in
vfio_mmcan be correctly removed when the mm is being destroyed, since we don't hold a reference of mm here.
yeah, I'll add it.
Quoted 78 lines
Quoted 78 lines
drivers/vfio/vfio.c +46
-0+return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);drivers/vfio/vfio_iommu_type1.c +2
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };struct vfio_domain {@@ -2018,6 +2019,7 @@ static void vfio_iommu_type1_detach_group(void *iommu_data, static void *vfio_iommu_type1_open(unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ;
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned · line 2051
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +0
-0iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );drivers/vfio/vfio_iommu_type1.c +3
-0if (!iommu)@@ -2043,6 +2045,10 @@ static void *vfio_iommu_type1_open(unsigned long arg) iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );I assume error should be returned when
pr_erris used...
got it. I didn't do it as I don't think vfio_mm is necessary for every iommu open. It is necessary for the nesting type iommu. I'll make it fetch vmm when it is opening nesting type and return error if failed.
Quoted 19 lines
Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu;}
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void *iommu_data) }vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu);}
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(structvfio_iommu *iommu, return ret; }
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2183
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+static bool vfio_iommu_type1_pasid_req_valid (u32 flags)I don't think you need prefix "
vfio_iommu_type1" for every new function here, especially for leaf internal function as this one.
got it. thanks.
Quoted 10 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc (struct vfio_iommu *iommu, +int min, +int max) +{
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2199
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;why
-EFAULT?
well, it's from a prior comment as below:
vfio_mm_pasid_alloc() can return -ENOSPC though, so it'd be nice to differentiate the errors. We could use EFAULT for the no IOMMU case and EINVAL here?
http://lkml.iu.edu/hypermail/linux/kernel/2001.3/05964.html
Quoted 57 lines
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ goto out_unlock; + } +if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +}drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2219
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +static int vfio_iommu_type1_pasid_free (struct vfio_iommu *iommu, +unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;ditto
Quoted 33 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ goto out_unlock; + } + +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg){
drivers/vfio/vfio_iommu_type1.c +13
-0@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, return copy_to_user((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz)) +return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags))
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;why copying in two steps instead of copying them together?
just want to do sanity check before copying all the data. I can move it as one copy if it's better. :-)
Quoted 16 lines
Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +15
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret = copy_to_user( + (void __user *) (arg + offset),
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2373
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ &result, sizeof(result)); + }else { +pr_debug ("%s: PASID alloc failed\n" , __func__); + ret = -EFAULT ;no, this branch is not for
copy_to_usererror. it is about pasid alloc failure. you should handle both.
Emmm, I just want to fail the IOCTL in such case, so the @result field is meaningless in the user side. How about using another return value (e.g. ENOSPC) to indicate the IOCTL failure?
Quoted 67 lines
Quoted 67 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+ } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + }drivers/vfio/vfio_iommu_type1.c +0
-0}return -ENOTTY ;include/linux/vfio.h +20
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */
include/uapi/linux/vfio.h +15
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { +__u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0) +#define VFIO_IOMMU_PASID_FREE (1 << 1)
Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 817
Quoted 6 lines
include/uapi/linux/vfio.h +6
-0+__u32 flags; + union { + struct { + __u32 min; + __u32 max; + __u32 result;
result->pasid?
yes, the pasid allocated.
Quoted 5 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 819
Quoted 3 lines
include/uapi/linux/vfio.h +2
-0+ } alloc_pasid; +__u32 free_pasid;what about putting a common pasid field after flags?
looks good to me. But it would make the union part only meaningful to alloc pasid. if so, maybe make the union part as a data field and only alloc pasid will have it. For free pasid, it is not necessary to read it from userspace. does it look good?
Regards,
Yi Liu
Quoted 24 lines
Quoted 24 lines
include/uapi/linux/vfio.h +18
-0+ }; +}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC | \ + VFIO_IOMMU_PASID_FREE) + +/** +* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it +* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
--
2.7.4
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 187 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 10:37 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:32 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/8] vfio: Add
VFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 4 lines
Quoted 4 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMQuoted 8 lines
Quoted 6 lines
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity
are->isthanks.
Quoted 2 lines
Quoted 2 lines
DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the
PCIe
Quoted 8 lines
Quoted 4 lines
PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the
"address space" -> "address spaces"
"binding the" -> "binding each"
will correct both.
Quoted 6 lines
Quoted 6 lines
virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model,
which
Quoted 6 lines
Quoted 6 lines
means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 7 lines
Quoted 3 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown
PASID allocation/free request from the virtual IOMMU. Additionally, such"Additionally, because such"
Quoted 1 line
requests are intended to be invoked by QEMU or other applications
which
Quoted 2 lines
simplify to "intended to be invoked from userspace"
got it.
Quoted 8 lines
Quoted 6 lines
are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is
devices -> device
got it.
Quoted 14 lines
Quoted 14 lines
possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in anotherpatch
Quoted 44 lines
Quoted 44 lines
in this series.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +1
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamsondrivers/vfio/vfio.c +2
-0@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;Quoted 1 line
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device
Quoted 1 line
drivers/vfio/vfio.c +0
-0@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device*dev,
Quoted 7 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2135
Quoted 5 lines
drivers/vfio/vfio.c +1
-0enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier );/** +* VFIO_MM objects - create, release, get, put, search why capitalizing
vfio_mm?oops, it's not intended, will fix it.
Quoted 33 lines
Quoted 33 lines
drivers/vfio/vfio.c +31
-0+* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm- >ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} +Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2170
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true);do we need hold pasid lock here, since it attempts to destroy a set which might be referenced by
vfio_mm_pasid_free? or is there guarantee that such race won't happen?
Quoted 3 lines
Emmm, if considering the race between
ioasid_free_setandvfio_mm_pasid_free, I guess ioasid core should sequence the two operations with its internal lock. right?
I looked at below code in free path:
+ mutex_lock(&vmm->pasid_lock);
+ pdata = ioasid_find(vmm->ioasid_sid, pasid, NULL);
+ if (IS_ERR(pdata)) {
+ ret = PTR_ERR(pdata);
+ goto out_unlock;
+ }
+ ioasid_free(pasid);
+
+out_unlock:
+ mutex_unlock(&vmm->pasid_lock);
above implies that ioasid_find/free must be paired within the pasid_lock. Then if we don't hold pasid_lock above, ioasid_free_set could happen between find/free. I'm not sure whether this race would lead to real problem, but it doesn't look correct simply by looking at this file.
Quoted 149 lines
Quoted 45 lines
Quoted 45 lines
drivers/vfio/vfio.c +16
-0+mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ + kref_put_mutex(&vmm->kref, vfio_mm_release,&vfio.vfio_mm_lock);
drivers/vfio/vfio.c +27
-0+} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out:Quoted 8 lines
drivers/vfio/vfio.c
Quoted 2 lines
drivers/vfio/vfio.c +2
-0+mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm);I assume this has been discussed before, but from readability p.o.v it might be good to add a comment for this function to explain how the recording of mm in
vfio_mmcan be correctly removed when the mm is being destroyed, since we don't hold a reference of mm here.yeah, I'll add it.
Quoted 70 lines
Quoted 70 lines
drivers/vfio/vfio.c +46
-0+return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);drivers/vfio/vfio_iommu_type1.c +1
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static void
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0struct vfio_domain {@@ -2018,6 +2019,7 @@ static voidvfio_iommu_type1_detach_group(void
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +1
-0*iommu_data, static void *vfio_iommu_type1_open(unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );drivers/vfio/vfio_iommu_type1.c +0
-0if (!iommu)@@ -2043,6 +2045,10 @@ static void*vfio_iommu_type1_open(unsigned
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long arg) · line 2050
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0long arg) iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );I assume error should be returned when
pr_erris used...
Quoted 4 lines
got it. I didn't do it as I don’t think
vfio_mmis necessary for every iommu open. It is necessary for the nesting type iommu. I'll make it fetch vmm when it is opening nesting type and return error if failed.
sounds good.
Quoted 118 lines
Quoted 20 lines
Quoted 20 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu;}
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void *iommu_data) }vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu);}
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(structvfio_iommu *iommu, return ret; }Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2183
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+static bool vfio_iommu_type1_pasid_req_valid (u32 flags)I don't think you need prefix "
vfio_iommu_type1" for every new function here, especially for leaf internal function as this one.got it. thanks.
Quoted 10 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc (struct vfio_iommu *iommu, +int min, +int max) +{Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2199
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;why
-EFAULT?well, it's from a prior comment as below:
vfio_mm_pasid_alloc()can return-ENOSPCthough, so it'd be nice to differentiate the errors. We could use EFAULT for the no IOMMU case and EINVAL here?
http://lkml.iu.edu/hypermail/linux/kernel/2001.3/05964.htmlQuoted 57 lines
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ goto out_unlock; + } +if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +}drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2219
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +static int vfio_iommu_type1_pasid_free (struct vfio_iommu *iommu, +unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;ditto
Quoted 33 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ goto out_unlock; + } + +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg){
drivers/vfio/vfio_iommu_type1.c +13
-0@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, return copy_to_user((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz)) +return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags))Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;why copying in two steps instead of copying them together?
Quoted 2 lines
just want to do sanity check before copying all the data. I can move it as one copy if it's better. :-)
it's possible fine. I just saw you did same thing for other uapis.
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +15
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret = copy_to_user( + (void __user *) (arg + offset),Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2373
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ &result, sizeof(result)); + }else { +pr_debug ("%s: PASID alloc failed\n" , __func__); + ret = -EFAULT ;no, this branch is not for
copy_to_usererror. it is about pasid alloc failure. you should handle both.
Quoted 3 lines
Emmm, I just want to fail the IOCTL in such case, so the @result field is meaningless in the user side. How about using another return value (e.g. ENOSPC) to indicate the IOCTL failure?
If pasid_alloc fails, you return its result to userspace if copy_to_user fails, then return -EFAULT.
however, above you return -EFAULT for pasid_alloc failure, and then the number of not-copied bytes for copy_to_user.
Quoted 86 lines
Quoted 22 lines
Quoted 22 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+ } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + }drivers/vfio/vfio_iommu_type1.c +0
-0}return -ENOTTY ;include/linux/vfio.h +0
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops);Quoted 16 lines
Quoted 6 lines
include/linux/vfio.h +6
-0+#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm {include/linux/vfio.h extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
Quoted 10 lines
include/linux/vfio.h +10
-0+struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task(struct task_struct*task);
Quoted 30 lines
Quoted 30 lines
include/linux/vfio.h +4
-0+extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */
include/uapi/linux/vfio.h +15
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { +__u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0) +#define VFIO_IOMMU_PASID_FREE (1 << 1)Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 817
Quoted 6 lines
include/uapi/linux/vfio.h +6
-0+__u32 flags; + union { + struct { + __u32 min; + __u32 max; + __u32 result;
result->pasid?yes, the pasid allocated.
Quoted 5 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 819
Quoted 3 lines
include/uapi/linux/vfio.h +2
-0+ } alloc_pasid; +__u32 free_pasid;what about putting a common pasid field after flags?
Quoted 4 lines
looks good to me. But it would make the union part only meaningful to alloc pasid. if so, maybe make the union part as a data field and only alloc pasid will have it. For free pasid, it is not necessary to read it from userspace. does it look good?
maybe keeping the union is also OK, just with {min, max} for alloc. who knows whether more pasid ops will be added in the future which may require its specific union structure. 😊 putting pasid as a common field is reasonable because the whole cmd is for pasid.
Quoted 30 lines
Regards,
Yi LiuQuoted 4 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 823
Quoted 4 lines
include/uapi/linux/vfio.h +4
-0+ }; +}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC| \
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/vfio.h +8
-0+ VFIO_IOMMU_PASID_FREE) + +/** +* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap {
Quoted 8 lines
include/uapi/linux/vfio.h +6
-0+* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/vfio.h +0
-1/* -- 2.7.4
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 215 lines
From: Tian, Kevin <[email protected]>
Sent: Tuesday, March 31, 2020 1:41 PM
To: Liu, Yi L <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 192 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 10:37 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:32 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/8] vfio: Add
VFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 6 lines
Quoted 6 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
Quoted 7 lines
Quoted 5 lines
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity
are->isthanks.
Quoted 3 lines
Quoted 3 lines
DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the
PCIe
Quoted 9 lines
Quoted 5 lines
PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the
"address space" -> "address spaces"
"binding the" -> "binding each"
will correct both.
Quoted 7 lines
Quoted 7 lines
virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model,
which
Quoted 6 lines
Quoted 6 lines
means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 7 lines
Quoted 3 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such"Additionally, because such"
Quoted 1 line
requests are intended to be invoked by QEMU or other applications
which
Quoted 2 lines
simplify to "intended to be invoked from userspace"
got it.
Quoted 9 lines
Quoted 7 lines
are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is
devices -> device
got it.
Quoted 16 lines
Quoted 16 lines
possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in anotherpatch
Quoted 44 lines
Quoted 44 lines
in this series.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c index renamed from drivers/vfio/vfio.c +0
-0c848262..d13b483 100644
drivers/vfio/vfio.c +1
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamsondrivers/vfio/vfio.c +2
-0@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;Quoted 1 line
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device
Quoted 1 line
drivers/vfio/vfio.c +0
-0@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device*dev,
Quoted 7 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2135
Quoted 5 lines
drivers/vfio/vfio.c +1
-0enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier );/** +* VFIO_MM objects - create, release, get, put, search why capitalizing
vfio_mm?oops, it's not intended, will fix it.
Quoted 31 lines
Quoted 31 lines
drivers/vfio/vfio.c +29
-0+* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) { +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm- >ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +}Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2170
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+ +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) { +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true);do we need hold pasid lock here, since it attempts to destroy a set which might be referenced by
vfio_mm_pasid_free? or is there guarantee that such race won't happen?Quoted 3 lines
Emmm, if considering the race between
ioasid_free_setandvfio_mm_pasid_free, I guess ioasid core should sequence the two operations with its internal lock. right?I looked at below code in free path:
drivers/vfio/vfio.c +10
-0+mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock);
Quoted 4 lines
above implies that ioasid_find/free must be paired within the
pasid_lock. Then if we don't holdpasid_lockabove,ioasid_free_setcould happen between find/free. I'm not sure whether this race would lead to real problem, but it doesn't look correct simply by looking at this file.
Well, Jacob told me to remove the ioasid_find in another email as he believes ioasid core should be able to take care of it. and also need to be protected by lock. If so, does it look good? :-)
" [Jacob Pan] this might be better to put under ioasid code such that it
is under the ioasid lock. no one else can free the ioasid between find() and free(). e.g. ioasid_free(sid, pasid)
if sid == INVALID_IOASID_SET, then no set ownership checking. thoughts?"
Quoted 300 lines
Quoted 140 lines
Quoted 40 lines
Quoted 40 lines
drivers/vfio/vfio.c +14
-0+mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ static void +vfio_mm_release (struct kref *kref) { + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) { + kref_put_mutex(&vmm->kref, vfio_mm_release,&vfio.vfio_mm_lock);
drivers/vfio/vfio.c +24
-0+} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ static +void vfio_mm_get (struct vfio_mm *vmm) { +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) { +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ;Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2215
Quoted 3 lines
drivers/vfio/vfio.c +3
-0+out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm);I assume this has been discussed before, but from readability p.o.v it might be good to add a comment for this function to explain how the recording of mm in
vfio_mmcan be correctly removed when the mm is being destroyed, since we don't hold a reference of mm here.yeah, I'll add it.
Quoted 67 lines
Quoted 67 lines
drivers/vfio/vfio.c +44
-1+return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) { +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) { +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode(struct device *dev, umode_t *mode) @@ -2151,8 +2279,10 @@ static int __init vfio_init(void) idr_init(&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex a177bf2..331ceee 100644drivers/vfio/vfio_iommu_type1.c +1
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static void
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0struct vfio_domain {@@ -2018,6 +2019,7 @@ static voidvfio_iommu_type1_detach_group(void
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static void
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +1
-0*iommu_data, static void *vfio_iommu_type1_open(unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );if (!iommu)@@ -2043,6 +2045,10 @@ static void*vfio_iommu_type1_open(unsigned
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long arg) · line 2050
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0long arg) iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );I assume error should be returned when
pr_erris used...Quoted 4 lines
got it. I didn't do it as I don’t think
vfio_mmis necessary for every iommu open. It is necessary for the nesting type iommu. I'll make it fetch vmm when it is opening nesting type and return error if failed.sounds good.
Quoted 118 lines
Quoted 20 lines
Quoted 20 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu;}
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void *iommu_data) }vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu);}
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(structvfio_iommu *iommu, return ret; }Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2183
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+static bool vfio_iommu_type1_pasid_req_valid (u32 flags)I don't think you need prefix "
vfio_iommu_type1" for every new function here, especially for leaf internal function as this one.got it. thanks.
Quoted 10 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc (struct vfio_iommu *iommu, +int min, +int max) +{Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2199
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;why
-EFAULT?well, it's from a prior comment as below:
vfio_mm_pasid_alloc()can return-ENOSPCthough, so it'd be nice to differentiate the errors. We could use EFAULT for the no IOMMU case and EINVAL here?
http://lkml.iu.edu/hypermail/linux/kernel/2001.3/05964.htmlQuoted 57 lines
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ goto out_unlock; + } +if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +}drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2219
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +static int vfio_iommu_type1_pasid_free (struct vfio_iommu *iommu, +unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ;ditto
Quoted 33 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ goto out_unlock; + } + +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg){
drivers/vfio/vfio_iommu_type1.c +13
-0@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, return copy_to_user((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz)) +return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags))Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;why copying in two steps instead of copying them together?
Quoted 2 lines
just want to do sanity check before copying all the data. I can move it as one copy if it's better. :-)
it's possible fine. I just saw you did same thing for other uapis.
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +15
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret = copy_to_user( + (void __user *) (arg + offset),Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
no, this branch is not for
copy_to_usererror. it is about pasid alloc failure. you should handle both.Quoted 3 lines
Emmm, I just want to fail the IOCTL in such case, so the @result field is meaningless in the user side. How about using another return value (e.g. ENOSPC) to indicate the IOCTL failure?
Quoted 5 lines
If
pasid_allocfails, you return its result to userspace ifcopy_to_userfails, then return-EFAULT.however, above you return
-EFAULTforpasid_allocfailure, and then the number of not-copied bytes forcopy_to_user.
not quite get. Let me re-paste the code. :-)
142 lines not shown · raw message
Parts of this message are not shown here. View the raw message.
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 232 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 31, 2020 9:22 PMQuoted 5 lines
From: Tian, Kevin <[email protected]>
Sent: Tuesday, March 31, 2020 1:41 PM
To: Liu, Yi L <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v1 1/8] vfio: Add
VFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 21 lines
Quoted 21 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 10:37 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:32 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/8] vfio: Add
VFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 12 lines
Quoted 11 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
are->isthanks.
Quoted 2 lines
Quoted 2 lines
DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform
vendors
Quoted 126 lines
Quoted 126 lines
Quoted 1 line
Quoted 1 line
are utilizing the
PCIe
Quoted 9 lines
Quoted 5 lines
PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the
"address space" -> "address spaces"
"binding the" -> "binding each"
will correct both.
Quoted 7 lines
Quoted 7 lines
virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model,
which
Quoted 6 lines
Quoted 6 lines
means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 7 lines
Quoted 3 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such"Additionally, because such"
Quoted 1 line
requests are intended to be invoked by QEMU or other applications
which
Quoted 2 lines
simplify to "intended to be invoked from userspace"
got it.
Quoted 9 lines
Quoted 7 lines
are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is
devices -> device
got it.
Quoted 16 lines
Quoted 16 lines
possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in anotherpatch
Quoted 44 lines
Quoted 44 lines
in this series.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c index renamed from drivers/vfio/vfio.c +0
-0c848262..d13b483 100644
drivers/vfio/vfio.c +1
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamsondrivers/vfio/vfio.c +2
-0@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;Quoted 1 line
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device
Quoted 1 line
drivers/vfio/vfio.c +0
-0@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device*dev,
Quoted 7 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2135
Quoted 5 lines
drivers/vfio/vfio.c +1
-0enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier );/** +* VFIO_MM objects - create, release, get, put, search why capitalizing
vfio_mm?oops, it's not intended, will fix it.
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) { +struct vfio_mm *vmm;Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,
Quoted 8 lines
drivers/vfio/vfio.c +8
-0+struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations
Quoted 45 lines
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
drivers/vfio/vfio.c +17
-0*/ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm- >ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +}Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2170
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+ +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) { +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true);do we need hold pasid lock here, since it attempts to destroy a set which might be referenced by
vfio_mm_pasid_free? or is there guarantee that such race won't happen?Quoted 3 lines
Emmm, if considering the race between
ioasid_free_setandvfio_mm_pasid_free, I guess ioasid core should sequence the two operations with its internal lock. right?I looked at below code in free path:
drivers/vfio/vfio.c +10
-0+mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock);Quoted 2 lines
above implies that ioasid_find/free must be paired within the
pasid_lock. Then if we don't holdpasid_lockabove,ioasid_free_setcould happenbetween
Quoted 1 line
find/free. I'm not sure whether this race would lead to real problem, but it
doesn't
Quoted 1 line
look correct simply by looking at this file.
Well, Jacob told me to remove the
ioasid_findin another email as he believes ioasid core should be able to take care of it. and also need to be protected by lock. If so, does it look good? :-)
Quoted 6 lines
" [Jacob Pan] this might be better to put under ioasid code such that it
is under the ioasid lock. no one else can free the ioasid betweenfind()andfree().
e.g.ioasid_free(sid, pasid)
if sid ==INVALID_IOASID_SET, then no set ownership checking. thoughts?"
yes, that way looks better.
Quoted 332 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,
Quoted 9 lines
drivers/vfio/vfio.c +7
-0+mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ static void +vfio_mm_release (struct kref *kref) { + struct vfio_mm *vmm = container_of(kref, struct vfio_mm,kref);
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
drivers/vfio/vfio.c +7
-0+ +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) { + kref_put_mutex(&vmm->kref, vfio_mm_release,&vfio.vfio_mm_lock);
drivers/vfio/vfio.c +23
-0+} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ static +void vfio_mm_get (struct vfio_mm *vmm) { +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) { +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm))Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2215
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+ vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm);I assume this has been discussed before, but from readability p.o.v it might be good to add a comment for this function to explain how the recording of mm in
vfio_mmcan be correctly removed when the266 lines not shown · raw message
drivers/vfio/vfio_iommu_type1.c +10
-0+case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( + struct1 line not shown · raw message
drivers/vfio/vfio_iommu_type1.c +4
-0+ alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result));
Quoted 5 lines
5 lines not shown · raw message
119 lines not shown · raw message
Part of this message is not shown here. View the raw message.
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 34 lines
From: Tian, Kevin <[email protected]>
Sent: Wednesday, April 1, 2020 1:43 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 29 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 31, 2020 9:22 PMQuoted 5 lines
From: Tian, Kevin <[email protected]>
Sent: Tuesday, March 31, 2020 1:41 PM
To: Liu, Yi L <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v1 1/8] vfio: Add
VFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 20 lines
Quoted 20 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 10:37 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:32 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/8] vfio: Add
VFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 11 lines
Quoted 11 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity
[...]
Quoted 57 lines
Quoted 57 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc(iommu,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+
req.alloc_pasid.min,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+
req.alloc_pasid.max);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+if (result >0 ) { + offset = offsetof( + struct
vfio_iommu_type1_pasid_request,drivers/vfio/vfio_iommu_type1.c +3
-0+ alloc_pasid.result); + ret = copy_to_user( + (void __user *) (arg +offset),
Quoted 13 lines
Quoted 9 lines
Quoted 8 lines
Review
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+ &result, sizeof(result)); + }else { + pr_debug("%s: PASID alloc failed\n",
__func__);(quoted diff) +1
-0+ ret = -EFAULT;no, this branch is not for
copy_to_usererror. it is about pasid alloc failure. you should handle both.Quoted 3 lines
Emmm, I just want to fail the IOCTL in such case, so the @result field is meaningless in the user side. How about using another return value (e.g. ENOSPC) to indicate the IOCTL failure?
Quoted 5 lines
If
pasid_allocfails, you return its result to userspace ifcopy_to_userfails, then return-EFAULT.however, above you return
-EFAULTforpasid_allocfailure, and then the number of not-copied bytes forcopy_to_user.not quite get. Let me re-paste the code. :-)
drivers/vfio/vfio_iommu_type1.c +10
-0+case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( + struct
vfio_iommu_type1_pasid_request,drivers/vfio/vfio_iommu_type1.c +3
-0+ alloc_pasid.result); + ret = copy_to_user( + (void __user *) (arg + offset),
Quoted 8 lines
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ &result, sizeof(result));if
copy_to_userfailed, ret is the number of uncopied bytes and will be returned to userspace to indicate failure. userspace will not use the data in result field. perhaps, I should check the ret here and return-EFAULTor another errno, instead of return the number of un-copied bytes.here should return
-EFAULT.
got it. so if any failure, the return value of this ioctl is a -ERROR_VAL.
Quoted 9 lines
Quoted 7 lines
(quoted diff) +2
-0+ } else { + pr_debug("%s: PASID alloc failed\n",
__func__);(quoted diff) +1
-0+ ret = -EFAULT;if
vfio_iommu_type1_pasid_alloc()failed, no doubt, return-EFAULTto userspace to indicate failure.
pasid_allochas its own error types returned. why blindly replace it with-EFAULT?
right, should use its own error types.
Thanks,
Yi Liu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Liu, Yi LWho is going to use thse exports? Please submit them together with a driver actually using them.
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 7 lines
From: Christoph Hellwig <[email protected]>
Sent: Tuesday, March 31, 2020 3:54 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Who is going to use thse exports? Please submit them together with a driver actually using them.
Hi Hellwig,
These are exposed for SVA (Shared Virtual Addressing) usage in VMs. If say a driver who actually using them, it is the iommu driver running in guest. The flow is: guest iommu driver programs the virtual command interface and it traps to host. The virtual IOMMU device model lays in QEMU will utilize the exported ioctl to get PASIDs.
Here is iommu kernel driver patch which utilizes virtual command interface to request pasid alloc/free.
https://lkml.org/lkml/2020/3/20/1176
And, the below patch is one which utilizes the ioctl exported in this patch:
https://patchwork.kernel.org/patch/11464601/
Regards,
Yi Liu
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Christoph HellwigQuoted 7 lines
From: Christoph Hellwig <[email protected]>
Sent: Tuesday, March 31, 2020 3:54 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Who is going to use thse exports? Please submit them together with a driver actually using them.
Hi Hellwig,
Sorry, maybe I misunderstood your point. Do you mean the exported symbol below? They are used by the vfio_iommu_type1 driver which is a separate driver besides the vfio.ko driver.
+EXPORT_SYMBOL_GPL(vfio_mm_put);
+EXPORT_SYMBOL_GPL(vfio_mm_get_from_task);
+EXPORT_SYMBOL_GPL(vfio_mm_pasid_alloc);
+EXPORT_SYMBOL_GPL(vfio_mm_pasid_free);
Regards,
Yi Liu
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 13 lines
From: Liu, Yi L
Sent: Tuesday, March 31, 2020 4:33 PM
To: 'Christoph Hellwig' <[email protected]>
Subject: RE: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 8 lines
From: Christoph Hellwig <[email protected]>
Sent: Tuesday, March 31, 2020 3:54 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Who is going to use thse exports? Please submit them together with a driver actually using them.
the user of the symbols are already in this patch. sorry for the split answer..
Regards,
Yi Liu
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 11 lines · Yi L
Quoted 11 lines
From: Liu, Yi L
Sent: Tuesday, March 31, 2020 4:33 PM
To: 'Christoph Hellwig' <[email protected]>
Subject: RE: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 6 lines
From: Christoph Hellwig <[email protected]>
Sent: Tuesday, March 31, 2020 3:54 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Who is going to use thse exports? Please submit them together with a driver actually using them.
the user of the symbols are already in this patch. sorry for the split answer..
Thanks, sorry for the noise!
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Liu, Yi LHi Yi,
Quoted 13 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in
Quoted 8 lines
software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access.
It's worth noting that this applies to Intel VT-d with scalable mode, not IOMMUs that use one PASID space per VM
Quoted 77 lines
It would be safer to
let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +8
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2149
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret = ioasid_alloc_set((struct ioasid_set *) mm,
Hmm, either we need to change the token of ioasid_alloc_set() to "void *", or pass an actual ioasid_set struct, but this cast doesn't look good :)
As I commented on the IOASID series, I think we could embed a struct ioasid_set into vfio_mm, pass that struct to all other ioasid_* functions, and get rid of ioasid_sid.
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2158
Quoted 9 lines
drivers/vfio/vfio.c +9
-0+ VFIO_DEFAULT_PASID_QUOTA, &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm;
Why the intermediate token struct? Could we just store the mm_struct pointer within vfio_mm?
Thanks,
Jean
Quoted 7 lines
drivers/vfio/vfio.c +7
-0+ vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +}
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Hi Jean,
Quoted 30 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, April 2, 2020 9:53 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Hi Yi,
Quoted 13 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in
Quoted 8 lines
software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access.
Quoted 2 lines
It's worth noting that this applies to Intel VT-d with scalable mode, not IOMMUs that use one PASID space per VM
Oh yes. will add it.
Quoted 93 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum
Quoted 70 lines
It would be safer to
let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +3
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev, enum
vfio_notify_typetype,Quoted 8 lines
drivers/vfio/vfio.c +5
-0EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2149
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret = ioasid_alloc_set((struct ioasid_set *) mm,Hmm, either we need to change the token of
ioasid_alloc_set()to "void *", or pass an actualioasid_setstruct, but this cast doesn't look good :)
Quoted 3 lines
As I commented on the IOASID series, I think we could embed a struct
ioasid_setintovfio_mm, pass that struct to all other ioasid_* functions, and get rid ofioasid_sid.
I think change to "void *" is better as we needs the token to ensure all threads within a single VM share the same ioasid_set.
Quoted 4 lines
Quoted 4 lines
drivers/vfio/vfio.c +4
-0+ VFIO_DEFAULT_PASID_QUOTA, &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret);
Quoted 8 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2158
Quoted 5 lines
drivers/vfio/vfio.c +5
-0+ } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm;Why the intermediate token struct? Could we just store the
mm_structpointer withinvfio_mm?
Hmm, here we only want to use the pointer as a token, instead of using the structure behind the pointer. If store the mm_struct directly, may leave a space to further use its content, this is not good.
Regards,
Yi Liu
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 20 lines · Yi L
Quoted 20 lines
Quoted 6 lines
/**
+ *VFIO_MMobjects - create, release, get, put, search
+ * Caller of the function should have held vfio.vfio_mm_lock.
+ */drivers/vfio/vfio.c +2
-0+static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2149
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret = ioasid_alloc_set((struct ioasid_set *) mm,Hmm, either we need to change the token of
ioasid_alloc_set()to "void *", or pass an actualioasid_setstruct, but this cast doesn't look good :)
Quoted 6 lines
Quoted 3 lines
As I commented on the IOASID series, I think we could embed a struct
ioasid_setintovfio_mm, pass that struct to all other ioasid_* functions, and get rid ofioasid_sid.I think change to "void *" is better as we needs the token to ensure all threads within a single VM share the same
ioasid_set.
Don't they share the same vfio_mm?
Thanks,
Jean
Quoted 21 lines
Quoted 13 lines
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2158
Quoted 10 lines
drivers/vfio/vfio.c +9
-0+ VFIO_DEFAULT_PASID_QUOTA, &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm;Why the intermediate token struct? Could we just store the
mm_structpointer withinvfio_mm?Hmm, here we only want to use the pointer as a token, instead of using the structure behind the pointer. If store the
mm_structdirectly, may leave a space to further use its content, this is not good.Regards,
Yi Liu
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 25 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 8:40 PM
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 20 lines · Yi L
Quoted 20 lines
Quoted 5 lines
/**
+ *VFIO_MMobjects - create, release, get, put, search
+ * Caller of the function should have held vfio.vfio_mm_lock.
+ */drivers/vfio/vfio.c +1
-0+static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) {drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type, · line 2149
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret = ioasid_alloc_set((struct ioasid_set *) mm,Hmm, either we need to change the token of
ioasid_alloc_set()to "void *", or pass an actualioasid_setstruct, but this cast doesn't look good :)
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
As I commented on the IOASID series, I think we could embed a struct
ioasid_setintovfio_mm, pass that struct to all other ioasid_* functions, and get rid ofioasid_sid.I think change to "void *" is better as we needs the token to ensure all threads within a single VM share the same
ioasid_set.Don't they share the same
vfio_mm?
that's right. then both works well for me.
Regards,
Yi Liu
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Liu, Yi LQuoted 15 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such
Quoted 8 lines
devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
Quoted 197 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +128
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} + +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true); +mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ +kref_put_mutex (&vmm->kref, vfio_mm_release, &vfio.vfio_mm_lock); +} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm); +return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)
drivers/vfio/vfio.c static int __init vfio_init(void) · line 2288
Quoted 10 lines
drivers/vfio/vfio.c +2
-0@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);
Is vfio.c the right place for any of the above? It seems like it could all be in a separate vfio_pasid module, similar to our virqfd module.
Quoted 21 lines
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };struct vfio_domain {@@ -2018,6 +2019,7 @@ static void vfio_iommu_type1_detach_group(void *iommu_data,static void *vfio_iommu_type1_open (unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );if (!iommu)
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long arg) · line 2050
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +3
-0@@ -2043,6 +2045,10 @@ static void *vfio_iommu_type1_open(unsigned long arg) iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );
Doesn't this presume everyone is instantly running PASID capable hosts? Looks like a noisy support regression to me.
Quoted 26 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ iommu->vmm = vmm;return iommu; }@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void *iommu_data) }vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu); }@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu,return ret; } +static bool vfio_iommu_type1_pasid_req_valid (u32 flags) +{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc(struct vfio_iommu *iommu, + int min,
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2201
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ int max) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }
Non-iommu backed mdevs are excluded from this? Is this a matter of wiring the call out through the mdev parent device, or is this just possible?
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2221
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }
So if a container had an iommu backed device when the pasid was allocated, but it was removed, now they can't free it? Why do we need the check above?
Quoted 26 lines
drivers/vfio/vfio_iommu_type1.c +19
-0+ +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz))
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags)) +return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;
Huh? Why do we have argsz if we're going to assume this is here?
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; +
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2370
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result));
Again assuming argsz supports this.
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2372
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ }else { +pr_debug ("%s: PASID alloc failed\n" , __func__);
rate limit?
Quoted 66 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ ret = -EFAULT ; + } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + } }return -ENOTTY ;include/linux/vfio.h +20
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */ include/uapi/linux/vfio.h +14
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { +__u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0)
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 820
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+#define VFIO_IOMMU_PASID_FREE (1 << 1) +__u32 flags; +union { +struct { +__u32 min; +__u32 max; +__u32 result; + } alloc_pasid; +__u32 free_pasid; + };
We seem to be using __u8 data[] lately where the struct at data is defined by the flags. should we do that here?
Quoted 6 lines
include/uapi/linux/vfio.h +6
-0+}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC | \ + VFIO_IOMMU_PASID_FREE) + +/**
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 836
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it +* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22)
So a user needs to try to allocate a PASID in order to test for the support? Should we have a PROBE flag?
Quoted 4 lines
include/uapi/linux/vfio.h +1
-0+/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AMQuoted 8 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the
PCIe
Quoted 7 lines
PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such
Quoted 8 lines
devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 3 lines
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
An user can access only PASIDs allocated to itself, i.e. the specific IOASID set tied to its mm_struct.
Thanks
Kevin
Quoted 496 lines
Quoted 2 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims topassdown
Quoted 35 lines
PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.
This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio.c | 130++++++++++++++++++++++++++++++++++++++++
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c | 104++++++++++++++++++++++++++++++++
Quoted 9 lines
include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +0
-0drivers/vfio/vfio.c line 38
Quoted 7 lines
drivers/vfio/vfio.c +1
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamsondrivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev,
Quoted 10 lines
drivers/vfio/vfio.c +2
-0@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev,enum
vfio_notify_typetype,Quoted 7 lines
drivers/vfio/vfio.c +4
-0EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm)drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */
Quoted 29 lines
drivers/vfio/vfio.c +28
-0+ ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm- ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} + +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true); +mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref)drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum vfio_notify_type type,
Quoted 10 lines
drivers/vfio/vfio.c +10
-0+{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ + kref_put_mutex(&vmm->kref, vfio_mm_release,&vfio.vfio_mm_lock);
Quoted 77 lines
drivers/vfio/vfio.c +73
-0+} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm); +return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode(struct device *dev,
umode_t*mode)drivers/vfio/vfio.c static int __init vfio_init(void) · line 2288
Quoted 10 lines
drivers/vfio/vfio.c +2
-0@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q); ret =misc_register (&vfio_dev);Is
vfio.cthe right place for any of the above? It seems like it could all be in a separatevfio_pasidmodule, similar to our virqfd module.Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cdrivers/vfio/vfio_iommu_type1.c static void
Quoted 12 lines
index a177bf2..331ceee 100644
drivers/vfio/vfio_iommu_type1.c +1
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };struct vfio_domain {@@ -2018,6 +2019,7 @@ static voidvfio_iommu_type1_detach_group(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +1
-0static void *vfio_iommu_type1_open (unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );if (!iommu)@@ -2043,6 +2045,10 @@ static void *vfio_iommu_type1_open(unsignedlong arg)
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long arg) · line 2050
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +3
-0iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );Doesn't this presume everyone is instantly running PASID capable hosts? Looks like a noisy support regression to me.
drivers/vfio/vfio_iommu_type1.c static void vfio_iommu_type1_release(void
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu; }@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void
*iommu_data)drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +2
-0}vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu); }@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(struct
vfio_iommu*iommu,Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +9
-0return ret; } +static bool vfio_iommu_type1_pasid_req_valid (u32 flags) +{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc(struct vfio_iommu *iommu, + int min,drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2201
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ int max) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }Non-iommu backed mdevs are excluded from this? Is this a matter of wiring the call out through the mdev parent device, or is this just possible?
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2221
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }So if a container had an iommu backed device when the pasid was allocated, but it was removed, now they can't free it? Why do we need the check above?
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 14 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +9
-0return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz))drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags)) +return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;Huh? Why do we have argsz if we're going to assume this is here?
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) { + offset = offsetof( + struct
vfio_iommu_type1_pasid_request,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2370
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result));Again assuming argsz supports this.
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void · line 2371
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ }else { + pr_debug("%s: PASID alloc failed\n",
__func__);rate limit?
include/linux/vfio.h extern int vfio_register_iommu_driver(const struct
Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ ret = -EFAULT ; + } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + } }return -ENOTTY ;include/linux/vfio.h +0
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct
vfio_iommu_driver_ops*ops);Quoted 48 lines
include/linux/vfio.h +20
-0extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */ include/uapi/linux/vfio.h +14
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { +__u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0)include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 820
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+#define VFIO_IOMMU_PASID_FREE (1 << 1) +__u32 flags; +union { +struct { +__u32 min; +__u32 max; +__u32 result; + } alloc_pasid; +__u32 free_pasid; + };We seem to be using
__u8data[] lately where the struct at data is defined by the flags. should we do that here?include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 823
Quoted 3 lines
include/uapi/linux/vfio.h +3
-0+}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC| \
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap {
Quoted 13 lines
include/uapi/linux/vfio.h +13
-0+ VFIO_IOMMU_PASID_FREE) + +/** +* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it +* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE +22)
So a user needs to try to allocate a PASID in order to test for the support? Should we have a PROBE flag?
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 838
Quoted 2 lines
include/uapi/linux/vfio.h +1
-0+/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */
Quoted 2 lines
include/uapi/linux/vfio.h +0
-0/*
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 31 lines · Kevin
Quoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AMQuoted 8 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the
PCIe
Quoted 15 lines
PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 6 lines
Quoted 3 lines
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
An user can access only PASIDs allocated to itself, i.e. the specific IOASID set tied to its
mm_struct.
A user is only _supposed_ to access PASIDs allocated to itself. AIUI the mm_struct is used for managing the pool of IOASIDs from which the user may allocate that PASID. We also state that programming the PASID into the device is device specific. Therefore, are we simply trusting the user to use a PASID that's been allocated to them when they program the device? If a user can program an arbitrary PASID into the device, then what prevents them from attempting to access data from another
user via the device? I think I've asked this question before, so if
there's a previous explanation or spec section I need to review, please point me to it. Thanks,
Alex
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 49 lines
From: Alex Williamson
Sent: Friday, April 3, 2020 11:14 PMQuoted 9 lines · Kevin
Quoted 9 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AMQuoted 3 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from
platform
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the
PCIe
Quoted 1 line
PASID TLP prefix in DMA requests, thus to give devices with multiple
DMA
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner.
Devices
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the
guest
Quoted 1 line
Quoted 1 line
Quoted 1 line
software could reuse the bare metal software programming model,
which
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 6 lines
Quoted 3 lines
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
An user can access only PASIDs allocated to itself, i.e. the specific IOASID set tied to its
mm_struct.
Quoted 11 lines
A user is only _supposed_ to access PASIDs allocated to itself. AIUI the
mm_structis used for managing the pool of IOASIDs from which the user may allocate that PASID. We also state that programming the PASID into the device is device specific. Therefore, are we simply trusting the user to use a PASID that's been allocated to them when they program the device? If a user can program an arbitrary PASID into the device, then what prevents them from attempting to access data from another
user via the device? I think I've asked this question before, so if
there's a previous explanation or spec section I need to review, please point me to it. Thanks,
There are two scenarios:
(1) for PF/VF, the iommu driver maintains an individual PASID table per PDF. Although the PASID namespace is global, the per-BDF PASID table contains only valid entries for those PASIDs which are allocated to the mm_struct. The user is free to program arbitrary PASID into the assigned device, but using invalid PASIDs simply hit iommu fault.
(2) for mdev, multiple mdev instances share the same PASID table of the parent BDF. However, PASID programming is a privileged operation in multiplexing usage, thus must be mediated by mdev device driver. The mediation logic will guarantee that only allocated PASIDs are forwarded to the device.
Thanks
Kevin
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 69 lines · Kevin
Quoted 49 lines
From: Alex Williamson
Sent: Friday, April 3, 2020 11:14 PMQuoted 9 lines · Kevin
Quoted 9 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AMQuoted 3 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from
platform
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the
PCIe
Quoted 1 line
PASID TLP prefix in DMA requests, thus to give devices with multiple
DMA
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner.
Devices
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
which support PASID capability are called PASID-capable devices. If such devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the
guest
Quoted 1 line
Quoted 1 line
Quoted 1 line
software could reuse the bare metal software programming model,
which
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Quoted 6 lines
Quoted 3 lines
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
An user can access only PASIDs allocated to itself, i.e. the specific IOASID set tied to its
mm_struct.Quoted 11 lines
A user is only _supposed_ to access PASIDs allocated to itself. AIUI the
mm_structis used for managing the pool of IOASIDs from which the user may allocate that PASID. We also state that programming the PASID into the device is device specific. Therefore, are we simply trusting the user to use a PASID that's been allocated to them when they program the device? If a user can program an arbitrary PASID into the device, then what prevents them from attempting to access data from another
user via the device? I think I've asked this question before, so if
there's a previous explanation or spec section I need to review, please point me to it. Thanks,There are two scenarios:
(1) for PF/VF, the iommu driver maintains an individual PASID table per PDF. Although the PASID namespace is global, the per-BDF PASID table contains only valid entries for those PASIDs which are allocated to the
mm_struct. The user is free to program arbitrary PASID into the assigned device, but using invalid PASIDs simply hit iommu fault.
Quoted 5 lines
(2) for mdev, multiple mdev instances share the same PASID table of the parent BDF. However, PASID programming is a privileged operation in multiplexing usage, thus must be mediated by mdev device driver. The mediation logic will guarantee that only allocated PASIDs are forwarded to the device.
Thanks, I was confused about multiple tenants sharing a BDF when PASID programming to the device is device specific, and therefore not something we can virtualize. However, the solution is device specific virtualization via mdev. Thus, any time we're sharing a BDF between tenants, we must virtualize the PASID programming and therefore it must be an mdev device currently. If a tenant is the exclusive user of the BDF, then no virtualization of the PASID programming is required. I think it's clear now (again). Thanks,
Alex
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Alex WilliamsonHi Alex,
Quoted 242 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AM
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 15 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such
Quoted 8 lines
devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum
Quoted 65 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +3
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev, enum
vfio_notify_typetype,Quoted 141 lines
drivers/vfio/vfio.c +127
-0EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} + +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true); +mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ +kref_put_mutex (&vmm->kref, vfio_mm_release, &vfio.vfio_mm_lock); +} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm); +return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q);
Quoted 4 lines
drivers/vfio/vfio.c static int __init vfio_init(void) · line 2288
Quoted 1 line
drivers/vfio/vfio.c +0
-0ret =misc_register (&vfio_dev);Is
vfio.cthe right place for any of the above? It seems like it could all be in a separatevfio_pasidmodule, similar to our virqfd module.
I think it could be a separate vfio_pasid module. let me make it in next version if it's your preference. :-)
Quoted 23 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_iommu_type1_detach_group(void
Quoted 13 lines
drivers/vfio/vfio_iommu_type1.c +1
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };struct vfio_domain {@@ -2018,6 +2019,7 @@ static void vfio_iommu_type1_detach_group(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +1
-0static void *vfio_iommu_type1_open (unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );if (!iommu)@@ -2043,6 +2045,10 @@ static void *vfio_iommu_type1_open(unsigned longarg)
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long arg) · line 2050
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +3
-0iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );Doesn't this presume everyone is instantly running PASID capable hosts? Looks like a noisy support regression to me.
right, it is. Kevin also questioned this part, I'll refine it and avoid regression noisy.
Quoted 34 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_iommu_type1_release(void
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu; }@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void
*iommu_data)drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +2
-0}vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu); }@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(struct
vfio_iommu*iommu,Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +15
-0return ret; } +static bool vfio_iommu_type1_pasid_req_valid (u32 flags) +{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc (struct vfio_iommu *iommu, +int min, +int max) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock);
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2201
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }Non-iommu backed mdevs are excluded from this? Is this a matter of wiring the call out through the mdev parent device, or is this just possible?
At the beginning, non-iommu backed mdevs are excluded. However, Combined with your succeeded comment. I think this check should be removed as the PASID alloc/free capability should be available as long as the container is backed by a pasid-capable iommu backend. So should remove it, and it is the same with the free path.
Quoted 17 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static int vfio_iommu_type1_pasid_free (struct vfio_iommu *iommu, +unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock);
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2201
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }So if a container had an iommu backed device when the pasid was allocated, but it was removed, now they can't free it? Why do we need the check above?
should be removed. thanks for spotting it.
Quoted 31 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 14 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +13
-0return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz)) +return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags))
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;Huh? Why do we have argsz if we're going to assume this is here?
do you mean replacing sizeof(req) with argsz? if yes, I can do that.
Quoted 10 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) {
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2370
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result));Again assuming argsz supports this.
same as above.
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2372
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ }else { +pr_debug ("%s: PASID alloc failed\n" , __func__);rate limit?
not quite get. could you give more hints?
Quoted 72 lines
include/linux/vfio.h extern int vfio_register_iommu_driver(const struct
Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ ret = -EFAULT ; + } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + } }return -ENOTTY ;include/linux/vfio.h +0
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct
vfio_iommu_driver_ops*ops);Quoted 53 lines
include/linux/vfio.h +20
-0extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */ include/uapi/linux/vfio.h +19
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { + __u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0) +#define VFIO_IOMMU_PASID_FREE (1 << 1) + __u32 flags; + union { + struct { + __u32 min;
Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 820
Quoted 5 lines
include/uapi/linux/vfio.h +5
-0+__u32 max; +__u32 result; + } alloc_pasid; +__u32 free_pasid; + };We seem to be using
__u8data[] lately where the struct at data is defined by the flags. should we do that here?
yeah, I can do that. BTW. Do you want to let the structure in the lately patch share the same structure with this one? As I can foresee, the two structures would look like similar as both of them include argsz, flags and data[] fields. The difference is the definition of flags. what about your opinion?
struct vfio_iommu_type1_pasid_request {
__u32 argsz;
#define VFIO_IOMMU_PASID_ALLOC (1 << 0)
#define VFIO_IOMMU_PASID_FREE (1 << 1)
__u32 flags;
__u8 data[];
};struct vfio_iommu_type1_bind {
__u32 argsz;
__u32 flags;
#define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0)
#define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1)
__u8 data[];
};Quoted 12 lines
Quoted 12 lines
include/uapi/linux/vfio.h +11
-0+}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC | \ + VFIO_IOMMU_PASID_FREE) + +/** +* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it
Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 836
Quoted 5 lines
include/uapi/linux/vfio.h +5
-0+* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22)So a user needs to try to allocate a PASID in order to test for the support? Should we have a PROBE flag?
answered in in later patch. :-)
Regards,
Yi Liu
Re: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 405 lines · Yi L
Hi Alex,
Quoted 242 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AM
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 15 lines · Yi L
From: Liu Yi L <[email protected]>
For a long time, devices have only one DMA address space from platform IOMMU's point of view. This is true for both bare metal and directed- access in virtualization environment. Reason is the source ID of DMA in PCIe are BDF (bus/dev/fnc ID), which results in only device granularity DMA isolation. However, this is changing with the latest advancement in I/O technology area. More and more platform vendors are utilizing the PCIe PASID TLP prefix in DMA requests, thus to give devices with multiple DMA address spaces as identified by their individual PASIDs. For example, Shared Virtual Addressing (SVA, a.k.a Shared Virtual Memory) is able to let device access multiple process virtual address space by binding the virtual address space with a PASID. Wherein the PASID is allocated in software and programmed to device per device specific manner. Devices which support PASID capability are called PASID-capable devices. If such
Quoted 8 lines
devices are passed through to VMs, guest software are also able to bind guest process virtual address space on such devices. Therefore, the guest software could reuse the bare metal software programming model, which means guest software will also allocate PASID and program it to device directly. This is a dangerous situation since it has potential PASID conflicts and unauthorized address space access. It would be safer to let host intercept in the guest software's PASID allocation. Thus PASID are managed system-wide.
Providing an allocation interface only allows for collaborative usage of PASIDs though. Do we have any ability to enforce PASID usage or can a user spoof other PASIDs on the same BDF?
drivers/vfio/vfio.c int vfio_unregister_notifier(struct device *dev, enum
Quoted 65 lines
This patch adds
VFIO_IOMMU_PASID_REQUESTioctl which aims to passdown PASID allocation/free request from the virtual IOMMU. Additionally, such requests are intended to be invoked by QEMU or other applications which are running in userspace, it is necessary to have a mechanism to prevent single application from abusing available PASIDs in system. With such consideration, this patch tracks the VFIO PASID allocation per-VM. There was a discussion to make quota to be per assigned devices. e.g. if a VM has many assigned devices, then it should have more quota. However, it is not sure how many PASIDs an assigned devices will use. e.g. it is possible that a VM with multiples assigned devices but requests less PASIDs. Therefore per-VM quota would be better.This patch uses struct mm pointer as a per-VM token. We also considered using task structure pointer and
vfio_iommustructure pointer. However, task structure is per-thread, which means it cannot achieve per-VM PASID alloc tracking purpose. While forvfio_iommustructure, it is visible only within vfio. Therefore, structure mm pointer is selected. This patch adds a structurevfio_mm. Avfio_mmis created when the first vfio container is opened by a VM. On the reverse order,vfio_mmis free when the last vfio container is released. Each VM is assigned with a PASID quota, so that it is not able to request PASID beyond its quota. This patch adds a default quota of 1000. This quota could be tuned by administrator. Making PASID quota tunable will be added in another patch in this series.Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Yi Sun <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio.c | 130 ++++++++++++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++ include/linux/vfio.h | 20 +++++++ include/uapi/linux/vfio.h | 41 +++++++++++++ 4 files changed, 295 insertions(+)drivers/vfio/vfio.c +3
-0@@ -32,6 +32,7 @@#include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/sched/mm.h> #define DRIVER_VERSION "0.3"#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"@@ -46,6 +47,8 @@ static struct vfio {struct mutex group_lock;struct cdev group_cdev;dev_t group_devt; +struct list_head vfio_mm_list; +struct mutex vfio_mm_lock;wait_queue_head_t release_q; } vfio;@@ -2129,6 +2132,131 @@ int vfio_unregister_notifier(struct device *dev, enum
vfio_notify_typetype,Quoted 141 lines
drivers/vfio/vfio.c +127
-0EXPORT_SYMBOL (vfio_unregister_notifier);/** +* VFIO_MM objects - create, release, get, put, search +* Caller of the function should have held vfio.vfio_mm_lock. +*/ +static struct vfio_mm *vfio_create_mm (struct mm_struct *mm) +{ +struct vfio_mm *vmm; +struct vfio_mm_token *token; +int ret =0 ; + + vmm =kzalloc (sizeof (*vmm),GFP_KERNEL ); +if (!vmm) +return ERR_PTR (-ENOMEM ); + +/* Per mm IOASID set used for quota control and group operations */ + ret =ioasid_alloc_set ((struct ioasid_set *) mm, +VFIO_DEFAULT_PASID_QUOTA , &vmm->ioasid_sid); +if (ret) { +kfree (vmm); +return ERR_PTR (ret); + } + +kref_init (&vmm->kref); + token = &vmm->token; + token->val = mm; + vmm->pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +mutex_init (&vmm->pasid_lock); + +list_add (&vmm->vfio_next, &vfio.vfio_mm_list); + +return vmm; +} + +static void vfio_mm_unlock_and_free (struct vfio_mm *vmm) +{ +/* destroy the ioasid set */ +ioasid_free_set (vmm->ioasid_sid, true); +mutex_unlock (&vfio.vfio_mm_lock); +kfree (vmm); +} + +/* called with vfio.vfio_mm_lock held */ +static void vfio_mm_release (struct kref *kref) +{ + struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref); + +list_del (&vmm->vfio_next); +vfio_mm_unlock_and_free (vmm); +} + +void vfio_mm_put (struct vfio_mm *vmm) +{ +kref_put_mutex (&vmm->kref, vfio_mm_release, &vfio.vfio_mm_lock); +} +EXPORT_SYMBOL_GPL (vfio_mm_put); + +/* Assume vfio_mm_lock or vfio_mm reference is held */ +static void vfio_mm_get (struct vfio_mm *vmm) +{ +kref_get (&vmm->kref); +} + +struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task) +{ +struct mm_struct *mm =get_task_mm (task); +struct vfio_mm *vmm; +unsigned long long val = (unsigned long long ) mm; + +mutex_lock (&vfio.vfio_mm_lock); +list_for_each_entry (vmm, &vfio.vfio_mm_list, vfio_next) { +if (vmm->token.val == val) { +vfio_mm_get (vmm); + goto out; + } + } + + vmm =vfio_create_mm (mm); +if (IS_ERR (vmm)) + vmm =NULL ; +out: +mutex_unlock (&vfio.vfio_mm_lock); +mmput (mm); +return vmm; +} +EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); + +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +{ +ioasid_t pasid; +int ret = -ENOSPC ; + +mutex_lock (&vmm->pasid_lock); + + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL ); +if (pasid ==INVALID_IOASID ) { + ret = -ENOSPC ; + goto out_unlock; + } + + ret = pasid; +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_alloc); + +int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid) +{ +void *pdata; +int ret =0 ; + +mutex_lock (&vmm->pasid_lock); + pdata =ioasid_find (vmm->ioasid_sid, pasid,NULL ); +if (IS_ERR (pdata)) { + ret =PTR_ERR (pdata); + goto out_unlock; + } +ioasid_free (pasid); + +out_unlock: +mutex_unlock (&vmm->pasid_lock); +return ret; +} +EXPORT_SYMBOL_GPL (vfio_mm_pasid_free); + +/** * Module/class support */ static char *vfio_devnode (struct device *dev,umode_t *mode)@@ -2151,8 +2279,10 @@ static int __init vfio_init(void)idr_init (&vfio.group_idr);mutex_init (&vfio.group_lock);mutex_init (&vfio.iommu_drivers_lock); +mutex_init (&vfio.vfio_mm_lock);INIT_LIST_HEAD (&vfio.group_list);INIT_LIST_HEAD (&vfio.iommu_drivers_list); +INIT_LIST_HEAD (&vfio.vfio_mm_list);init_waitqueue_head (&vfio.release_q);Quoted 4 lines
drivers/vfio/vfio.c static int __init vfio_init(void) · line 2288
Quoted 1 line
drivers/vfio/vfio.c +0
-0ret =misc_register (&vfio_dev);Is
vfio.cthe right place for any of the above? It seems like it could all be in a separatevfio_pasidmodule, similar to our virqfd module.I think it could be a separate
vfio_pasidmodule. let me make it in next version if it's your preference. :-)Quoted 23 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_iommu_type1_detach_group(void
Quoted 13 lines
drivers/vfio/vfio_iommu_type1.c +1
-0@@ -70,6 +70,7 @@ struct vfio_iommu {unsigned int dma_avail;bool v2;bool nesting; +struct vfio_mm *vmm; };struct vfio_domain {@@ -2018,6 +2019,7 @@ static void vfio_iommu_type1_detach_group(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +1
-0static void *vfio_iommu_type1_open (unsigned long arg) {struct vfio_iommu *iommu; +struct vfio_mm *vmm =NULL ; iommu =kzalloc (sizeof (*iommu),GFP_KERNEL );if (!iommu)@@ -2043,6 +2045,10 @@ static void *vfio_iommu_type1_open(unsigned longarg)
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void *vfio_iommu_type1_open(unsigned long arg) · line 2050
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +3
-0iommu->dma_avail = dma_entry_limit;mutex_init (&iommu->lock);BLOCKING_INIT_NOTIFIER_HEAD (&iommu->notifier); + vmm =vfio_mm_get_from_task (current); +if (!vmm) +pr_err ("Failed to get vfio_mm track\n" );Doesn't this presume everyone is instantly running PASID capable hosts? Looks like a noisy support regression to me.
right, it is. Kevin also questioned this part, I'll refine it and avoid regression noisy.
Quoted 34 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_iommu_type1_release(void
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ iommu->vmm = vmm;return iommu; }@@ -2084,6 +2090,8 @@ static void vfio_iommu_type1_release(void
*iommu_data)drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +2
-0}vfio_iommu_iova_free (&iommu->iova_list); +if (iommu->vmm) +vfio_mm_put (iommu->vmm);kfree (iommu); }@@ -2172,6 +2180,55 @@ static int vfio_iommu_iova_build_caps(struct
vfio_iommu*iommu,Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +15
-0return ret; } +static bool vfio_iommu_type1_pasid_req_valid (u32 flags) +{ +return !((flags & ~VFIO_PASID_REQUEST_MASK ) || + (flags &VFIO_IOMMU_PASID_ALLOC && + flags &VFIO_IOMMU_PASID_FREE )); +} + +static int vfio_iommu_type1_pasid_alloc (struct vfio_iommu *iommu, +int min, +int max) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2201
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }Non-iommu backed mdevs are excluded from this? Is this a matter of wiring the call out through the mdev parent device, or is this just possible?
At the beginning, non-iommu backed mdevs are excluded. However, Combined with your succeeded comment. I think this check should be removed as the PASID alloc/free capability should be available as long as the container is backed by a pasid-capable iommu backend. So should remove it, and it is the same with the free path.
Quoted 17 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+if (vmm) + ret =vfio_mm_pasid_alloc (vmm, min, max); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static int vfio_iommu_type1_pasid_free (struct vfio_iommu *iommu, +unsigned int pasid) +{ +struct vfio_mm *vmm = iommu->vmm; +int ret =0 ; + +mutex_lock (&iommu->lock);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, · line 2201
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EFAULT ; + goto out_unlock; + }So if a container had an iommu backed device when the pasid was allocated, but it was removed, now they can't free it? Why do we need the check above?
should be removed. thanks for spotting it.
Quoted 31 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 14 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +if (vmm) + ret =vfio_mm_pasid_free (vmm, pasid); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +13
-0return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz)) +return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags))Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;Huh? Why do we have argsz if we're going to assume this is here?
Quoted 1 line
do you mean replacing sizeof(req) with argsz? if yes, I can do that.
No, I mean the user tells us how much data they've provided via argsz. We create minsz the the end of flags and verify argsz includes flags. Then we proceed to ignore argsz to see if the user has provided the remainder of the structure.
Quoted 22 lines
Quoted 10 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) {Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2370
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result));Again assuming argsz supports this.
same as above.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2372
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ }else { +pr_debug ("%s: PASID alloc failed\n" , __func__);rate limit?
not quite get. could you give more hints?
A user can spam the host logs simply by allocating their quota of PASIDs and then trying to allocate more, or by specifying min/max such that they cannot allocate the requested PASID. If this logging is necessary for debugging, it should be ratelimited to avoid a DoS on the host.
Quoted 95 lines
Quoted 72 lines
include/linux/vfio.h extern int vfio_register_iommu_driver(const struct
Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ ret = -EFAULT ; + } +return ret; + } +case VFIO_IOMMU_PASID_FREE : +return vfio_iommu_type1_pasid_free (iommu, + req.free_pasid); +default : +return -EINVAL ; + } }return -ENOTTY ;include/linux/vfio.h +0
-0@@ -89,6 +89,26 @@ extern int vfio_register_iommu_driver(const struct
vfio_iommu_driver_ops*ops);Quoted 53 lines
include/linux/vfio.h +20
-0extern void vfio_unregister_iommu_driver (const struct vfio_iommu_driver_ops *ops); +#define VFIO_DEFAULT_PASID_QUOTA 1000 +struct vfio_mm_token { +unsigned long long val; +}; + +struct vfio_mm { +struct kref kref; +struct vfio_mm_token token; +int ioasid_sid; +/* protect @pasid_quota field and pasid allocation/free */ +struct mutex pasid_lock; +int pasid_quota; +struct list_head vfio_next; +}; + +extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task); +extern void vfio_mm_put (struct vfio_mm *vmm); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid); +/* * External user API */ include/uapi/linux/vfio.h +19
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { + __u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0) +#define VFIO_IOMMU_PASID_FREE (1 << 1) + __u32 flags; + union { + struct { + __u32 min;Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 820
Quoted 5 lines
include/uapi/linux/vfio.h +5
-0+__u32 max; +__u32 result; + } alloc_pasid; +__u32 free_pasid; + };We seem to be using
__u8data[] lately where the struct at data is defined by the flags. should we do that here?yeah, I can do that. BTW. Do you want to let the structure in the lately patch share the same structure with this one? As I can foresee, the two structures would look like similar as both of them include argsz, flags and data[] fields. The difference is the definition of flags. what about your opinion?
struct vfio_iommu_type1_pasid_request { __u32 argsz; #define VFIO_IOMMU_PASID_ALLOC (1 << 0) #define VFIO_IOMMU_PASID_FREE (1 << 1) __u32 flags; __u8 data[]; };
Quoted 7 lines
struct vfio_iommu_type1_bind {
__u32 argsz;
__u32 flags;
#define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0)
#define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1)
__u8 data[];
};Yes, I was even wondering the same for the cache invalidate ioctl, or whether this is going too far for a general purpose "everything related to PASIDs" ioctl. We need to factor usability into the equation too. I'd be interested in opinions from others here too. Clearly I don't like single use, throw-away ioctls, but I can find myself on either side of the argument that allocation, binding, and invalidating are all within the domain of PASIDs and could fall within a single ioctl or they each represent different facets of managing PASIDs and should have separate ioctls. Thanks,
Alex
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
include/uapi/linux/vfio.h +11
-0+}; + +#define VFIO_PASID_REQUEST_MASK (VFIO_IOMMU_PASID_ALLOC | \ + VFIO_IOMMU_PASID_FREE) + +/** +* VFIO_IOMMU_PASID_REQUEST - _IOWR(VFIO_TYPE, VFIO_BASE + 22, +* struct vfio_iommu_type1_pasid_request) +* +* Availability of this feature depends on PASID support in the device, +* its bus, the underlying IOMMU and the CPU architecture. In VFIO, it Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 836
Quoted 5 lines
include/uapi/linux/vfio.h +5
-0+* is available after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22)So a user needs to try to allocate a PASID in order to test for the support? Should we have a PROBE flag?
answered in in later patch. :-)
Regards,
Yi Liu
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
Quoted 2 lines
From: Alex Williamson
Sent: Saturday, April 4, 2020 1:50 AM
[...]
Quoted 59 lines
Quoted 50 lines
Quoted 27 lines
Quoted 27 lines
include/uapi/linux/vfio.h +19
-0@@ -794,6 +794,47 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/* +* PASID (Process Address Space ID) is a PCIe concept which +* has been extended to support DMA isolation in fine-grain. +* With device assigned to user space (e.g. VMs), PASID alloc +* and free need to be system wide. This structure defines +* the info for pasid alloc/free between user space and kernel +* space. +* +* @flag=VFIO_IOMMU_PASID_ALLOC, refer to the @alloc_pasid +* @flag=VFIO_IOMMU_PASID_FREE, refer to @free_pasid +*/ +struct vfio_iommu_type1_pasid_request { + __u32 argsz; +#define VFIO_IOMMU_PASID_ALLOC (1 << 0) +#define VFIO_IOMMU_PASID_FREE (1 << 1) + __u32 flags; + union { + struct { + __u32 min;Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 820
Quoted 5 lines
include/uapi/linux/vfio.h +5
-0+__u32 max; +__u32 result; + } alloc_pasid; +__u32 free_pasid; + };We seem to be using
__u8data[] lately where the struct at data is defined by the flags. should we do that here?yeah, I can do that. BTW. Do you want to let the structure in the lately patch share the same structure with this one? As I can foresee, the two structures would look like similar as both of them include argsz, flags and data[] fields. The difference is the definition of flags. what about your opinion?
struct vfio_iommu_type1_pasid_request { __u32 argsz; #define VFIO_IOMMU_PASID_ALLOC (1 << 0) #define VFIO_IOMMU_PASID_FREE (1 << 1) __u32 flags; __u8 data[]; };Quoted 7 lines
struct vfio_iommu_type1_bind { __u32 argsz; __u32 flags; #define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0) #define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1) __u8 data[]; };
Quoted 10 lines
Yes, I was even wondering the same for the cache invalidate ioctl, or whether this is going too far for a general purpose "everything related to PASIDs" ioctl. We need to factor usability into the equation too. I'd be interested in opinions from others here too. Clearly I don't like single use, throw-away ioctls, but I can find myself on either side of the argument that allocation, binding, and invalidating are all within the domain of PASIDs and could fall within a single ioctl or they each represent different facets of managing PASIDs and should have separate ioctls. Thanks,
Looking at uapi/linux/iommu.h:
* Invalidations by %IOMMU_INV_GRANU_DOMAIN don't take any argument other than
* @version and @cache.
Although intel-iommu handles only PASID-related invalidation now, I suppose other vendors (or future usages?) may allow non-pasid based invalidation too based on above comment.
Thanks
Kevin
RE: [PATCH v1 1/8] vfio: Add VFIO_IOMMU_PASID_REQUEST(alloc/free)
In reply to Alex WilliamsonHi Alex,
Quoted 18 lines
From: Alex Williamson <[email protected]>
Sent: Saturday, April 4, 2020 1:50 AM
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 11 lines · Yi L
Hi Alex,
Quoted 9 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:50 AM
Subject: Re: [PATCH v1 1/8] vfio: AddVFIO_IOMMU_PASID_REQUEST(alloc/free)Quoted 2 lines · Yi L
From: Liu Yi L <[email protected]>
[...]
Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 4 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +0
-0unsigned int cmd, unsigned long arg) {@@ -2276,6 +2333,53 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +9
-0return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + + }else if (cmd ==VFIO_IOMMU_PASID_REQUEST ) { +struct vfio_iommu_type1_pasid_request req; +unsigned long offset; + + minsz = offsetofend(struct vfio_iommu_type1_pasid_request, + flags); + +if (copy_from_user (&req, (void __user *)arg, minsz))drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2354
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+return -EFAULT ; + +if (req.argsz < minsz || + !vfio_iommu_type1_pasid_req_valid (req.flags)) +return -EINVAL ; + +if (copy_from_user ((void *)&req + minsz, + (void __user *)arg + minsz, +sizeof (req) - minsz)) +return -EFAULT ;Huh? Why do we have argsz if we're going to assume this is here?
Quoted 6 lines
Quoted 1 line
do you mean replacing sizeof(req) with argsz? if yes, I can do that.
No, I mean the user tells us how much data they've provided via argsz. We create minsz the the end of flags and verify argsz includes flags. Then we proceed to ignore argsz to see if the user has provided the remainder of the structure.
I think I should avoid using sizeof(req) as it may be variable new flag is added. I think better to make a data[] field in struct vfio_iommu_type1_pasid_request and copy data[] per flag. I'll make this change in new version.
Quoted 27 lines
Quoted 27 lines
Quoted 10 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +switch (req.flags &VFIO_PASID_REQUEST_MASK ) { +case VFIO_IOMMU_PASID_ALLOC : + { +int ret =0 , result; + + result =vfio_iommu_type1_pasid_alloc (iommu, + req.alloc_pasid.min, + req.alloc_pasid.max); +if (result >0 ) {Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2370
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ offset = offsetof( +struct vfio_iommu_type1_pasid_request , + alloc_pasid.result); + ret =copy_to_user ( + (void __user *) (arg + offset), + &result,sizeof (result));Again assuming argsz supports this.
same as above.
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2372
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ }else { +pr_debug ("%s: PASID alloc failed\n" , __func__);rate limit?
Quoted 7 lines
Quoted 1 line
not quite get. could you give more hints?
A user can spam the host logs simply by allocating their quota of PASIDs and then trying to allocate more, or by specifying min/max such that they cannot allocate the requested PASID. If this logging is necessary for debugging, it should be ratelimited to avoid a DoS on the host.
got it. thanks for the coaching. will use pr_debug_ratelimited().
Regards,
Yi Liu
PATCH 2/8 vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-process.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/
--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)
drivers/vfio/vfio.c +7 -1
@@ -2217,13 +2217,19 @@ struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min,int max) {ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock); +/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } + pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL );if (pasid ==INVALID_IOASID ) { ret = -ENOSPC ;
drivers/vfio/vfio_iommu_type1.c +6 -1
@@ -60,6 +60,11 @@ module_param_named(dma_entry_limit, dma_entry_limit, uint, 0644);MODULE_PARM_DESC (dma_entry_limit,"Maximum number of user DMA mappings per container (65535)." ); +static int pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +module_param_named (pasid_quota, pasid_quota, uint,0644 ); +MODULE_PARM_DESC (pasid_quota, +"Quota of user owned PASIDs per vfio-based application (1000)." ); +struct vfio_iommu {struct list_head domain_list;struct list_head iova_list;
@@ -2200,7 +2205,7 @@ static int vfio_iommu_type1_pasid_alloc(struct vfio_iommu *iommu, goto out_unlock; }if (vmm) - ret =vfio_mm_pasid_alloc (vmm, min, max); + ret =vfio_mm_pasid_alloc (vmm, pasid_quota, min, max);else ret = -EINVAL ; out_unlock:
include/linux/vfio.h +2 -1
@@ -106,7 +106,8 @@ struct vfio_mm {extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task);extern void vfio_mm_put (struct vfio_mm *vmm); -extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm, +int quota,int min,int max);extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid);/*
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
kbuild test robot
build botRe: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Thank you for the patch! Yet something to improve:
Hi Yi,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vfio/next]
[also build test ERROR on v5.6-rc6 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Liu-Yi-L/vfio-expose-virtual-Shared-Virtual-Addressing-to-VMs/20200322-213259 base: https://github.com/awilliam/linux-vfio.git next config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
drivers/vfio/vfio.c: In function 'vfio_create_mm':
drivers/vfio/vfio.c:2149:8: error: implicit declaration of function 'ioasid_alloc_set'; did you mean 'ioasid_alloc'? [-Werror=implicit-function-declaration]
2149 | ret = ioasid_alloc_set((struct ioasid_set *) mm,
| ^~~~~~~~~~~~~~~~
| ioasid_alloc
drivers/vfio/vfio.c:2158:13: warning: assignment to 'long long unsigned int' from 'struct mm_struct *' makes integer from pointer without a cast [-Wint-conversion]
2158 | token->val = mm;
| ^
drivers/vfio/vfio.c: In function 'vfio_mm_unlock_and_free':
drivers/vfio/vfio.c:2170:2: error: implicit declaration of function 'ioasid_free_set'; did you mean 'ioasid_free'? [-Werror=implicit-function-declaration]
2170 | ioasid_free_set(vmm->ioasid_sid, true);
| ^~~~~~~~~~~~~~~
| ioasid_free
drivers/vfio/vfio.c: In function 'vfio_mm_pasid_alloc':
>> drivers/vfio/vfio.c:2230:3: error: implicit declaration of function 'ioasid_adjust_set' [-Werror=implicit-function-declaration]
2230 | ioasid_adjust_set(vmm->ioasid_sid, quota);
| ^~~~~~~~~~~~~~~~~
drivers/vfio/vfio.c:2233:26: warning: passing argument 1 of 'ioasid_alloc' makes pointer from integer without a cast [-Wint-conversion]
2233 | pasid = ioasid_alloc(vmm->ioasid_sid, min, max, NULL);
| ~~~^~~~~~~~~~~~
| |
| int
In file included from include/linux/iommu.h:16,
from drivers/vfio/vfio.c:20:
include/linux/ioasid.h:45:56: note: expected 'struct ioasid_set *' but argument is of type 'int'
45 | static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
| ~~~~~~~~~~~~~~~~~~~^~~
drivers/vfio/vfio.c: In function 'vfio_mm_pasid_free':
drivers/vfio/vfio.c:2252:25: warning: passing argument 1 of 'ioasid_find' makes pointer from integer without a cast [-Wint-conversion]
2252 | pdata = ioasid_find(vmm->ioasid_sid, pasid, NULL);
| ~~~^~~~~~~~~~~~
| |
| int In file included from include/linux/iommu.h:16,
from drivers/vfio/vfio.c:20:
include/linux/ioasid.h:55:52: note: expected 'struct ioasid_set *' but argument is of type 'int'
55 | static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
| ~~~~~~~~~~~~~~~~~~~^~~
cc1: some warnings being treated as errors
vim +/ioasid_adjust_set +2230 drivers/vfio/vfio.c
2133
2134 /**
2135 * VFIO_MM objects - create, release, get, put, search
2136 * Caller of the function should have held vfio.vfio_mm_lock.
2137 */
2138 static struct vfio_mm *vfio_create_mm(struct mm_struct *mm)
2139 {
2140 struct vfio_mm *vmm;
2141 struct vfio_mm_token *token;
2142 int ret = 0;
2143
2144 vmm = kzalloc(sizeof(*vmm), GFP_KERNEL);
2145 if (!vmm)
2146 return ERR_PTR(-ENOMEM);
2147
2148 /* Per mm IOASID set used for quota control and group operations */
2149 ret = ioasid_alloc_set((struct ioasid_set *) mm,
2150 VFIO_DEFAULT_PASID_QUOTA, &vmm->ioasid_sid);
2151 if (ret) {
2152 kfree(vmm);
2153 return ERR_PTR(ret);
2154 }
2155
2156 kref_init(&vmm->kref);
2157 token = &vmm->token;
> 2158 token->val = mm;
2159 vmm->pasid_quota = VFIO_DEFAULT_PASID_QUOTA;
2160 mutex_init(&vmm->pasid_lock);
2161
2162 list_add(&vmm->vfio_next, &vfio.vfio_mm_list);
2163
2164 return vmm;
2165 }
2166
2167 static void vfio_mm_unlock_and_free(struct vfio_mm *vmm)
2168 {
2169 /* destroy the ioasid set */
2170 ioasid_free_set(vmm->ioasid_sid, true);
2171 mutex_unlock(&vfio.vfio_mm_lock);
2172 kfree(vmm);
2173 }
2174
2175 /* called with vfio.vfio_mm_lock held */
2176 static void vfio_mm_release(struct kref *kref)
2177 {
2178 struct vfio_mm *vmm = container_of(kref, struct vfio_mm, kref);
2179
2180 list_del(&vmm->vfio_next);
2181 vfio_mm_unlock_and_free(vmm);
2182 }
2183
2184 void vfio_mm_put(struct vfio_mm *vmm)
2185 {
2186 kref_put_mutex(&vmm->kref, vfio_mm_release, &vfio.vfio_mm_lock);
2187 }
2188 EXPORT_SYMBOL_GPL(vfio_mm_put);
2189
2190 /* Assume vfio_mm_lock or vfio_mm reference is held */
2191 static void vfio_mm_get(struct vfio_mm *vmm)
2192 {
2193 kref_get(&vmm->kref);
2194 }
2195
2196 struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task)
2197 {
2198 struct mm_struct *mm = get_task_mm(task);
2199 struct vfio_mm *vmm;
2200 unsigned long long val = (unsigned long long) mm;
2201
2202 mutex_lock(&vfio.vfio_mm_lock);
2203 list_for_each_entry(vmm, &vfio.vfio_mm_list, vfio_next) {
2204 if (vmm->token.val == val) {
2205 vfio_mm_get(vmm);
2206 goto out;
2207 }
2208 }
2209
2210 vmm = vfio_create_mm(mm);
2211 if (IS_ERR(vmm))
2212 vmm = NULL;
2213 out:
2214 mutex_unlock(&vfio.vfio_mm_lock);
2215 mmput(mm);
2216 return vmm;
2217 }
2218 EXPORT_SYMBOL_GPL(vfio_mm_get_from_task);
2219
2220 int vfio_mm_pasid_alloc(struct vfio_mm *vmm, int quota, int min, int max)
2221 {
2222 ioasid_t pasid;
2223 int ret = -ENOSPC;
2224
2225 mutex_lock(&vmm->pasid_lock);
2226
2227 /* update quota as it is tunable by admin */
2228 if (vmm->pasid_quota != quota) {
2229 vmm->pasid_quota = quota;
> 2230 ioasid_adjust_set(vmm->ioasid_sid, quota); 2231 }
2232
2233 pasid = ioasid_alloc(vmm->ioasid_sid, min, max, NULL);
2234 if (pasid == INVALID_IOASID) {
2235 ret = -ENOSPC;
2236 goto out_unlock;
2237 }
2238
2239 ret = pasid;
2240 out_unlock:
2241 mutex_unlock(&vmm->pasid_lock);
2242 return ret;
2243 }
2244 EXPORT_SYMBOL_GPL(vfio_mm_pasid_alloc);
2245
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RE: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
In reply to Liu, Yi LQuoted 38 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-process.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)drivers/vfio/vfio.c +1
-1@@ -2217,13 +2217,19 @@ struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min,int max) {ioasid_t pasid;
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct · line 2233
Quoted 10 lines
drivers/vfio/vfio.c +6
-0int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock); +/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +
It's a bit weird to have quota adjusted in the alloc path, since the latter might be initiated by non-privileged users. Why not doing the simple math in vfio_ create_mm to set the quota when the ioasid set is created? even in the future you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc...
Quoted 49 lines
drivers/vfio/vfio.c +0
-0pasid =ioasid_alloc (vmm->ioasid_sid, min, max,NULL );if (pasid ==INVALID_IOASID ) {drivers/vfio/vfio.c +0
-0ret = -ENOSPC ;drivers/vfio/vfio_iommu_type1.c +5
-0@@ -60,6 +60,11 @@ module_param_named(dma_entry_limit, dma_entry_limit, uint, 0644);MODULE_PARM_DESC (dma_entry_limit, "Maximum number of user DMA mappings per container (65535)."); +static int pasid_quota =VFIO_DEFAULT_PASID_QUOTA ; +module_param_named (pasid_quota, pasid_quota, uint,0644 ); +MODULE_PARM_DESC (pasid_quota, + "Quota of user owned PASIDs per vfio-based application (1000)."); +struct
vfio_iommu{drivers/vfio/vfio_iommu_type1.c +1
-1struct list_head domain_list;struct list_head iova_list;@@ -2200,7 +2205,7 @@ static int vfio_iommu_type1_pasid_alloc(structvfio_iommu *iommu, goto out_unlock; }if (vmm) - ret =vfio_mm_pasid_alloc (vmm, min, max); + ret =vfio_mm_pasid_alloc (vmm, pasid_quota, min, max);else ret = -EINVAL ;
out_unlock:include/linux/vfio.h +2
-2@@ -106,7 +106,8 @@ struct vfio_mm {extern struct vfio_mm *vfio_mm_get_from_task (struct task_struct *task);extern void vfio_mm_put (struct vfio_mm *vmm); -extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max); +extern int vfio_mm_pasid_alloc (struct vfio_mm *vmm, +int quota,int min,int max);extern int vfio_mm_pasid_free (struct vfio_mm *vmm,ioasid_t pasid);/* -- 2.7.4
RE: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Quoted 48 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:41 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameter for quota tuningQuoted 42 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-process.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)drivers/vfio/vfio.c +1
-1@@ -2217,13 +2217,19 @@ struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min,int max) {ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock);
Quoted 12 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) · line 2232
Quoted 6 lines
drivers/vfio/vfio.c +6
-0+/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +It's a bit weird to have quota adjusted in the alloc path, since the latter might be initiated by non-privileged users. Why not doing the simple math in
vfio_create_mmto set the quota when the ioasid set is created? even in the future you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc..
The reason is the kernel parameter modification has no event which can be used to adjust the quota. So I chose to adjust it in pasid_alloc path. If it's not good, how about adding one more IOCTL to let user- space trigger a quota adjustment event? Then even non-privileged user could trigger quota adjustment, the quota is actually controlled by privileged user. How about your opinion?
Regards,
Yi Liu
RE: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Quoted 70 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 4:53 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:41 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 27 lines
tuning
Quoted 25 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-process.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)Quoted 5 lines
drivers/vfio/vfio.c struct vfio_mm
Quoted 5 lines
drivers/vfio/vfio.c +0
-0@@ -2217,13 +2217,19 @@ struct vfio_mm*vfio_mm_get_from_task(struct
Quoted 6 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task)
Quoted 6 lines
drivers/vfio/vfio.c +1
-1task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc(struct vfio_mm *vmm, int quota, int min, intmax)
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio.c +0
-0{ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock);Quoted 8 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) · line 2232
Quoted 6 lines
drivers/vfio/vfio.c +6
-0+/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +It's a bit weird to have quota adjusted in the alloc path, since the latter
might
Quoted 1 line
be initiated by non-privileged users. Why not doing the simple math in
vfio_Quoted 1 line
create_mmto set the quota when the ioasid set is created? even in thefuture
Quoted 2 lines
you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc..
Quoted 7 lines
The reason is the kernel parameter modification has no event which can be used to adjust the quota. So I chose to adjust it in
pasid_allocpath. If it's not good, how about adding one more IOCTL to let user- space trigger a quota adjustment event? Then even non-privileged user could trigger quota adjustment, the quota is actually controlled by privileged user. How about your opinion?
why do you need an event to adjust? As I said, you can set the quota when the set is created in vfio_create_mm...
Thanks
Kevin
RE: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Quoted 89 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 5:20 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameter for quota tuningQuoted 75 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 4:53 PMQuoted 5 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:41 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 27 lines
tuning
Quoted 25 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-process.
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)Quoted 5 lines
drivers/vfio/vfio.c struct vfio_mm
Quoted 5 lines
drivers/vfio/vfio.c index renamed from drivers/vfio/vfio.c +0
-0d13b483..020a792 100644
drivers/vfio/vfio.c +0
-0@@ -2217,13 +2217,19 @@ struct vfio_mm*vfio_mm_get_from_task(struct
Quoted 7 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task)
Quoted 7 lines
drivers/vfio/vfio.c +2
-1task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min, +intmax)
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio.c +0
-0{ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock);Quoted 9 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) · line 2232
Quoted 6 lines
drivers/vfio/vfio.c +6
-0+/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +It's a bit weird to have quota adjusted in the alloc path, since the latter
might
Quoted 2 lines
be initiated by non-privileged users. Why not doing the simple math in
vfio_Quoted 2 lines
create_mmto set the quota when the ioasid set is created? even in thefuture
Quoted 2 lines
you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc..
Quoted 7 lines
The reason is the kernel parameter modification has no event which can be used to adjust the quota. So I chose to adjust it in
pasid_allocpath. If it's not good, how about adding one more IOCTL to let user- space trigger a quota adjustment event? Then even non-privileged user could trigger quota adjustment, the quota is actually controlled by privileged user. How about your opinion?
Quoted 2 lines
why do you need an event to adjust? As I said, you can set the quota when the set is created in
vfio_create_mm...
oh, it's to support runtime adjustments. I guess it may be helpful to let per-VM quota tunable even the VM is running. If just set the quota in vfio_create_mm(), it is not able to adjust at runtime.
Regards,
Yi Liu
RE: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Quoted 94 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 5:27 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 5:20 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 22 lines
tuning
Quoted 20 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 4:53 PMQuoted 5 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:41 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 11 lines
tuning
Quoted 9 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-
process.
Quoted 63 lines
Quoted 62 lines
Quoted 16 lines
Quoted 16 lines
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)Quoted 5 lines
drivers/vfio/vfio.c struct vfio_mm
Quoted 5 lines
drivers/vfio/vfio.c index renamed from drivers/vfio/vfio.c +0
-0d13b483..020a792 100644
drivers/vfio/vfio.c +0
-0@@ -2217,13 +2217,19 @@ struct vfio_mm*vfio_mm_get_from_task(struct
Quoted 7 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task)
Quoted 7 lines
drivers/vfio/vfio.c +2
-1task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min, +intmax)
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio.c +0
-0{ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock);Quoted 9 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) · line 2232
Quoted 6 lines
drivers/vfio/vfio.c +6
-0+/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +It's a bit weird to have quota adjusted in the alloc path, since the latter
might
Quoted 2 lines
be initiated by non-privileged users. Why not doing the simple math in
vfio_Quoted 2 lines
create_mmto set the quota when the ioasid set is created? even in thefuture
Quoted 2 lines
you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc..
The reason is the kernel parameter modification has no event which can be used to adjust the quota. So I chose to adjust it in
pasid_allocpath. If it's not good, how about adding one more IOCTL to let user- space trigger a quota adjustment event? Then even non-privileged user could trigger quota adjustment, the quota is actually controlled by privileged user. How about your opinion?
Quoted 8 lines
Quoted 1 line
why do you need an event to adjust? As I said, you can set the quota when
the set is
Quoted 1 line
created in
vfio_create_mm...oh, it's to support runtime adjustments. I guess it may be helpful to let per-VM quota tunable even the VM is running. If just set the quota in
vfio_create_mm(), it is not able to adjust at runtime.
ok, I didn't note the module parameter was granted with a write permission. However there is a further problem. We cannot support PASID reclaim now. What about the admin sets a quota smaller than previous value while some IOASID sets already exceed the new quota? I'm not sure how to fail a runtime module parameter change due to that situation. possibly a normal sysfs node better suites the runtime change requirement...
Thanks
Kevin
Re: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Quoted 103 lines · Kevin
Quoted 94 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 5:27 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 5:20 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 22 lines
tuning
Quoted 20 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 4:53 PMQuoted 5 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:41 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 11 lines
tuning
Quoted 9 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-
process.
Quoted 63 lines
Quoted 62 lines
Quoted 16 lines
Quoted 16 lines
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)Quoted 5 lines
drivers/vfio/vfio.c struct vfio_mm
Quoted 5 lines
drivers/vfio/vfio.c index renamed from drivers/vfio/vfio.c +0
-0d13b483..020a792 100644
drivers/vfio/vfio.c +0
-0@@ -2217,13 +2217,19 @@ struct vfio_mm*vfio_mm_get_from_task(struct
Quoted 7 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task)
Quoted 7 lines
drivers/vfio/vfio.c +2
-1task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min, +intmax)
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio.c +0
-0{ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock);Quoted 9 lines
drivers/vfio/vfio.c struct vfio_mm *vfio_mm_get_from_task(struct task_struct *task) · line 2232
Quoted 6 lines
drivers/vfio/vfio.c +6
-0+/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +It's a bit weird to have quota adjusted in the alloc path, since the latter
might
Quoted 2 lines
be initiated by non-privileged users. Why not doing the simple math in
vfio_Quoted 2 lines
create_mmto set the quota when the ioasid set is created? even in thefuture
Quoted 2 lines
you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc..
The reason is the kernel parameter modification has no event which can be used to adjust the quota. So I chose to adjust it in
pasid_allocpath. If it's not good, how about adding one more IOCTL to let user- space trigger a quota adjustment event? Then even non-privileged user could trigger quota adjustment, the quota is actually controlled by privileged user. How about your opinion?Quoted 8 lines
Quoted 1 line
why do you need an event to adjust? As I said, you can set the quota when
the set is
Quoted 1 line
created in
vfio_create_mm...oh, it's to support runtime adjustments. I guess it may be helpful to let per-VM quota tunable even the VM is running. If just set the quota in
vfio_create_mm(), it is not able to adjust at runtime.
Quoted 6 lines
ok, I didn't note the module parameter was granted with a write permission. However there is a further problem. We cannot support PASID reclaim now. What about the admin sets a quota smaller than previous value while some IOASID sets already exceed the new quota? I'm not sure how to fail a runtime module parameter change due to that situation. possibly a normal sysfs node better suites the runtime change requirement...
Yep, making this runtime adjustable seems a bit unpredictable and racy, and it's not clear to me how a user is going to jump in at just the right time for a user and adjust the limit. I'd probably go for a simple non-runtime adjustable module option. It's a safety net at this point anyway afaict. Thanks,
Alex
RE: [PATCH v1 2/8] vfio/type1: Add vfio_iommu_type1 parameter for quota tuning
Quoted 119 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 1:59 AM
To: Tian, Kevin <[email protected]>
Subject: Re: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameter for quota tuningQuoted 103 lines · Kevin
Quoted 94 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 5:27 PMQuoted 4 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 5:20 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 22 lines
tuning
Quoted 20 lines
From: Liu, Yi L <[email protected]>
Sent: Monday, March 30, 2020 4:53 PMQuoted 5 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 4:41 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 2/8] vfio/type1: Addvfio_iommu_type1parameterfor quota
Quoted 11 lines
tuning
Quoted 9 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch adds a module option to make the PASID quota tunable by administrator.
TODO: needs to think more on how to make the tuning to be per-
process.
Quoted 63 lines
Quoted 62 lines
Quoted 16 lines
Quoted 16 lines
Previous discussions:
https://patchwork.kernel.org/patch/11209429/Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio.c | 8 +++++++- drivers/vfio/vfio_iommu_type1.c | 7 ++++++- include/linux/vfio.h | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)Quoted 5 lines
drivers/vfio/vfio.c struct vfio_mm
Quoted 5 lines
drivers/vfio/vfio.c index renamed from drivers/vfio/vfio.c +0
-0d13b483..020a792 100644
drivers/vfio/vfio.c +0
-0@@ -2217,13 +2217,19 @@ struct vfio_mm*vfio_mm_get_from_task(struct
Quoted 7 lines
drivers/vfio/vfio.c struct vfio_mm · line 2222
Quoted 7 lines
drivers/vfio/vfio.c +2
-1task_struct *task) }EXPORT_SYMBOL_GPL (vfio_mm_get_from_task); -int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int min,int max) +int vfio_mm_pasid_alloc (struct vfio_mm *vmm,int quota,int min, +intmax)
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio.c +0
-0{ioasid_t pasid;int ret = -ENOSPC ;mutex_lock (&vmm->pasid_lock);Quoted 9 lines
drivers/vfio/vfio.c struct vfio_mm · line 2234
Quoted 6 lines
drivers/vfio/vfio.c +6
-0+/* update quota as it is tunable by admin */ +if (vmm->pasid_quota != quota) { + vmm->pasid_quota = quota; +ioasid_adjust_set (vmm->ioasid_sid, quota); + } +It's a bit weird to have quota adjusted in the alloc path, since the latter
might
Quoted 2 lines
be initiated by non-privileged users. Why not doing the simple math in
vfio_Quoted 2 lines
create_mmto set the quota when the ioasid set is created? even in thefuture
Quoted 2 lines
you may allow per-process quota setting, that should come from separate privileged path instead of thru alloc..
The reason is the kernel parameter modification has no event which can be used to adjust the quota. So I chose to adjust it in
pasid_allocpath. If it's not good, how about adding one more IOCTL to let user- space trigger a quota adjustment event? Then even non-privileged user could trigger quota adjustment, the quota is actually controlled by privileged user. How about your opinion?Quoted 8 lines
Quoted 1 line
why do you need an event to adjust? As I said, you can set the quota when
the set is
Quoted 1 line
created in
vfio_create_mm...oh, it's to support runtime adjustments. I guess it may be helpful to let per-VM quota tunable even the VM is running. If just set the quota in
vfio_create_mm(), it is not able to adjust at runtime.Quoted 6 lines
ok, I didn't note the module parameter was granted with a write permission. However there is a further problem. We cannot support PASID reclaim now. What about the admin sets a quota smaller than previous value while some IOASID sets already exceed the new quota? I'm not sure how to fail a runtime module parameter change due to that situation. possibly a normal sysfs node better suites the runtime change requirement...
Quoted 5 lines
Yep, making this runtime adjustable seems a bit unpredictable and racy, and it's not clear to me how a user is going to jump in at just the right time for a user and adjust the limit. I'd probably go for a simple non-runtime adjustable module option. It's a safety net at this point anyway afaict. Thanks,
thanks, I can do the changes.
Regards,
Yi Liu
PATCH 3/8 vfio/type1: Report PASID alloc/free support to userspace
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)
drivers/vfio/vfio_iommu_type1.c +28 -0
@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ; +if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; + } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {
@@ -2283,6 +2307,10 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;
include/uapi/linux/vfio.h +8 -0
@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2 + +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0) +__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)/**
Signature · 1 line
2.7.4
RE: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
Quoted 37 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)drivers/vfio/vfio_iommu_type1.c +8
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(structvfio_iommu *iommu, return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header = vfio_info_cap_add(caps, sizeof(*nesting_cap), + VFIO_IOMMU_TYPE1_INFO_CAP_NESTING,
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct · line 2255
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +9
-01); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ; +if (iommu->nesting) {
Is it good to report a nesting cap when iommu->nesting is disabled? I suppose the check should move before vfio_info_cap_add...
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=
VFIO_IOMMU_PASID_REQS;
VFIO_IOMMU_CAP_PASID_REQ? to avoid confusion with ioctl cmd VFIO_IOMMU_PASID_REQUEST...
Quoted 41 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+ } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2283,6 +2307,10 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +8
-1@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2 + +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0) +__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)/** -- 2.7.4
RE: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
Quoted 47 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 5:44 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspaceQuoted 41 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex e40afc0..ddd1ffe 100644drivers/vfio/vfio_iommu_type1.c +12
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(structvfio_iommu *iommu, return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap = container_of(header,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2253
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ; +if (iommu->nesting) {Is it good to report a nesting cap when
iommu->nestingis disabled? I suppose the check should move beforevfio_info_cap_add...
oops, yes it.
Quoted 8 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=
VFIO_IOMMU_PASID_REQS;
VFIO_IOMMU_CAP_PASID_REQ? to avoid confusion with ioctl cmdVFIO_IOMMU_PASID_REQUEST...
got it.
Thanks,
Yi Liu
Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
In reply to Liu, Yi LYi,
Quoted 33 lines · Yi L
From: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)drivers/vfio/vfio_iommu_type1.c +9
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header))
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2255
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ; +if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;
Supporting nesting does not necessarily mean supporting PASID.
Quoted 24 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+ } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2283,6 +2307,10 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +0
-0
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 755
Quoted 8 lines
include/uapi/linux/vfio.h +5
-0@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2 + +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0)
PASID_REQS sounds a bit far from the claimed host managed alloc/free capability.VFIO_IOMMU_SYSTEM_WIDE_PASID?
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 761
Quoted 7 lines
include/uapi/linux/vfio.h +3
-0+__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)/**
Thanks
Eric
RE: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
Hi Eric,
Quoted 45 lines
From: Auger Eric <[email protected]>
Sent: Wednesday, April 1, 2020 5:41 PM
To: Liu, Yi L <[email protected]>; [email protected]
Subject: Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspaceYi,
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct
Quoted 21 lines · Yi L
From: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex e40afc0..ddd1ffe 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,Quoted 15 lines
drivers/vfio/vfio_iommu_type1.c +12
-0return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap = container_of(header,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2255
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ; +if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;Supporting nesting does not necessarily mean supporting PASID.
here I think the PASID is somehow IDs in kernel which can be used to tag various address spaces provided by guest software. I think this is why we introduced the ioasid. :-) Current implementation is doing PASID alloc/free in vfio.
Quoted 27 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2283,6 +2307,10 @@
Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +0
-0static long vfio_iommu_type1_ioctl(void *iommu_data,drivers/vfio/vfio_iommu_type1.c +4
-0if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +0
-0@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; };
Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 755
Quoted 5 lines
include/uapi/linux/vfio.h +5
-0+#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2 + +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0)
PASID_REQSsounds a bit far from the claimed host managed alloc/free capability.VFIO_IOMMU_SYSTEM_WIDE_PASID?
Oh, yep. I can rename it.
Regards,
Yi Liu
Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
In reply to Liu, Yi LQuoted 71 lines · Yi L
From: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)drivers/vfio/vfio_iommu_type1.c +28
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ; +if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; + } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2283,6 +2307,10 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +1
-0@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 761
Quoted 10 lines
include/uapi/linux/vfio.h +7
-0+ +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0) +__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)/**
I think this answers my PROBE question on patch 1/. Should the quota/usage be exposed to the user here? Thanks,
Alex
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
Quoted 90 lines
From: Alex Williamson < [email protected] >
Sent: Friday, April 3, 2020 2:01 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspacedrivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct
Quoted 21 lines · Yi L
From: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex e40afc0..ddd1ffe 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +16
-0return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ;drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; + } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2283,6 +2307,10 @@
Quoted 29 lines
drivers/vfio/vfio_iommu_type1.c +0
-0static long vfio_iommu_type1_ioctl(void *iommu_data,drivers/vfio/vfio_iommu_type1.c +4
-0if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +8
-0@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2 + +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0) +__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
Quoted 3 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 761
Quoted 1 line
include/uapi/linux/vfio.h +0
-0/** I think this answers my PROBE question on patch 1/.
yep.
Quoted 1 line
Should the quota/usage be exposed to the user here? Thanks,
Do you mean report the quota available for this user in this cap info as well? For usage, do you mean the alloc and free or others?
Regards,
Yi Liu
Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
Quoted 90 lines · Yi L
Quoted 90 lines
From: Alex Williamson < [email protected] >
Sent: Friday, April 3, 2020 2:01 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspacedrivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct
Quoted 21 lines · Yi L
From: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex e40afc0..ddd1ffe 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,6 +2234,30 @@ static int vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +16
-0return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ;drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; + } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2283,6 +2307,10 @@
Quoted 29 lines
drivers/vfio/vfio_iommu_type1.c +0
-0static long vfio_iommu_type1_ioctl(void *iommu_data,drivers/vfio/vfio_iommu_type1.c +4
-0if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +8
-0@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2 + +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0) +__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
Quoted 7 lines
Quoted 3 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 761
Quoted 1 line
include/uapi/linux/vfio.h +0
-0/** I think this answers my PROBE question on patch 1/.
yep.
Quoted 1 line
Should the quota/usage be exposed to the user here? Thanks,
Do you mean report the quota available for this user in this cap info as well?
Yes. Would it be useful?
Quoted 1 line
For usage, do you mean the alloc and free or others?
I mean how many of the quota are currently in allocated, or alternatively, how many remain. Thanks,
Alex
RE: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspace
Hi Alex,
Quoted 107 lines
From: Alex Williamson <[email protected]>
Sent: Saturday, April 4, 2020 1:28 AM
To: Liu, Yi L <[email protected]>
Cc: [email protected]; Tian, Kevin <[email protected]>; [email protected]; [email protected]; Raj, Ashok <[email protected]>; Tian, Jun J <[email protected]>; Sun, Yi Y <[email protected]>; jean- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Wu, Hao <[email protected]>
Subject: Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspaceQuoted 93 lines · Yi L
Quoted 93 lines
From: Alex Williamson < [email protected] >
Sent: Friday, April 3, 2020 2:01 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 3/8] vfio/type1: Report PASID alloc/free support to userspaceQuoted 21 lines · Yi L
From: Liu Yi L <[email protected]>
This patch reports PASID alloc/free availability to userspace (e.g. QEMU) thus userspace could do a pre-check before utilizing this feature.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 36 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex e40afc0..ddd1ffe 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,6 +2234,30 @@ static intQuoted 1 line
vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +15
-0return ret; } +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) { +struct vfio_info_cap_header *header; +struct vfio_iommu_type1_info_cap_nesting *nesting_cap; + + header =vfio_info_cap_add (caps,sizeof (*nesting_cap), +VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 ); +if (IS_ERR (header)) +return PTR_ERR (header); + + nesting_cap =container_of (header, + struct vfio_iommu_type1_info_cap_nesting, + header); + + nesting_cap->nesting_capabilities =0 ;drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+if (iommu->nesting) { +/* nesting iommu type supports PASID requests (alloc/free) */ + nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; + } + +return 0 ; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2283,6 +2307,10 @@
Quoted 20 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +4
-0if (ret)return ret; + ret =vfio_iommu_info_add_nesting_cap (iommu, &caps); +if (ret) +return ret; +if (caps.size) { info.flags |=VFIO_IOMMU_INFO_CAPS ;include/uapi/linux/vfio.h +1
-0@@ -748,6 +748,14 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +#define VFIO_IOMMU_TYPE1_INFO_CAP_NESTING 2include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_iova_range { · line 761
Quoted 10 lines
include/uapi/linux/vfio.h +7
-0+ +struct vfio_iommu_type1_info_cap_nesting { +struct vfio_info_cap_header header; +#define VFIO_IOMMU_PASID_REQS (1 << 0) +__u32 nesting_capabilities; +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)/** I think this answers my PROBE question on patch 1/.
Quoted 6 lines
Quoted 4 lines
yep.
Quoted 1 line
Should the quota/usage be exposed to the user here? Thanks,
Do you mean report the quota available for this user in this cap info as well?
Yes. Would it be useful?
I think so.
Quoted 4 lines
Quoted 1 line
For usage, do you mean the alloc and free or others?
I mean how many of the quota are currently in allocated, or alternatively, how many remain. Thanks,
ok, got it, maybe report the remain. thanks.
Regards,
Yi Liu
PATCH 4/8 vfio: Check nesting iommu uAPI version
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
In Linux Kernel, the IOMMU nesting translation (a.k.a dual stage address translation) capability is abstracted in uapi/iommu.h, in which the uAPIs like bind_gpasid/iommu_cache_invalidate/fault_report/pgreq_resp are defined.
VFIO_TYPE1_NESTING_IOMMU stands for the vfio iommu type which is backed by hardware IOMMU w/ dual stage translation capability. For such vfio iommu type, userspace is able to setup dual stage DMA translation in host side via VFIO's ABI. However, such VFIO ABIs rely on the uAPIs defined in uapi/ iommu.h. So VFIO needs to provide an API to userspace for the uapi/iommu.h version check to ensure the iommu uAPI compatibility.
This patch reports the iommu uAPI version to userspace in VFIO_CHECK_EXTENSION IOCTL. Applications could do version check before further setup dual stage translation in host IOMMU.
--- drivers/vfio/vfio_iommu_type1.c | 2 ++ include/uapi/linux/vfio.h | 9 +++++++++ 2 files changed, 11 insertions(+)
drivers/vfio/vfio_iommu_type1.c +2 -0
@@ -2274,6 +2274,8 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,if (!iommu)return 0 ;return vfio_domains_have_iommu_cache (iommu); +case VFIO_NESTING_IOMMU_UAPI : +return iommu_get_uapi_version ();default :return 0 ; }
include/uapi/linux/vfio.h +9 -0
@@ -47,6 +47,15 @@#define VFIO_NOIOMMU_IOMMU 8/* +* Hardware IOMMUs with two-stage translation capability give userspace +* the ownership of stage-1 translation structures (e.g. page tables). +* VFIO exposes the two-stage IOMMU programming capability to userspace +* based on the IOMMU UAPIs. Therefore user of VFIO_TYPE1_NESTING should +* check the IOMMU UAPI version compatibility. +*/ +#define VFIO_NESTING_IOMMU_UAPI 9 + +/* * The IOCTL interface is designed for extensibility by embedding the * structure length (argsz) and flags into structures passed between * kernel and userspace. We therefore use the _IO() macro for these
Signature · 1 line
2.7.4
kbuild test robot
build botRe: [PATCH v1 4/8] vfio: Check nesting iommu uAPI version
Thank you for the patch! Yet something to improve:
Hi Yi,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vfio/next]
[also build test ERROR on v5.6-rc6 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Liu-Yi-L/vfio-expose-virtual-Shared-Virtual-Addressing-to-VMs/20200322-213259 base: https://github.com/awilliam/linux-vfio.git next config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
drivers/vfio/vfio_iommu_type1.c: In function 'vfio_iommu_type1_ioctl':
>> drivers/vfio/vfio_iommu_type1.c:2299:11: error: implicit declaration of function 'iommu_get_uapi_version' [-Werror=implicit-function-declaration]
2299 | return iommu_get_uapi_version();
| ^~~~~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/iommu_get_uapi_version +2299 drivers/vfio/vfio_iommu_type1.c
2281
2282 static long vfio_iommu_type1_ioctl(void *iommu_data,
2283 unsigned int cmd, unsigned long arg)
2284 {
2285 struct vfio_iommu *iommu = iommu_data;
2286 unsigned long minsz;
2287
2288 if (cmd == VFIO_CHECK_EXTENSION) {
2289 switch (arg) {
2290 case VFIO_TYPE1_IOMMU:
2291 case VFIO_TYPE1v2_IOMMU:
2292 case VFIO_TYPE1_NESTING_IOMMU:
2293 return 1;
2294 case VFIO_DMA_CC_IOMMU:
2295 if (!iommu)
2296 return 0;
2297 return vfio_domains_have_iommu_cache(iommu);
2298 case VFIO_NESTING_IOMMU_UAPI:
> 2299 return iommu_get_uapi_version();
2300 default:
2301 return 0;
2302 }
2303 } else if (cmd == VFIO_IOMMU_GET_INFO) {
2304 struct vfio_iommu_type1_info info;
2305 struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
2306 unsigned long capsz;
2307 int ret;
2308
2309 minsz = offsetofend(struct vfio_iommu_type1_info, iova_pgsizes);
2310
2311 /* For backward compatibility, cannot require this */
2312 capsz = offsetofend(struct vfio_iommu_type1_info, cap_offset);
2313
2314 if (copy_from_user(&info, (void __user *)arg, minsz))
2315 return -EFAULT;
2316
2317 if (info.argsz < minsz)
2318 return -EINVAL;
2319
2320 if (info.argsz >= capsz) {
2321 minsz = capsz;
2322 info.cap_offset = 0; /* output, no-recopy necessary */
2323 }
2324
2325 info.flags = VFIO_IOMMU_INFO_PGSIZES;
2326
2327 info.iova_pgsizes = vfio_pgsize_bitmap(iommu);
2328
2329 ret = vfio_iommu_iova_build_caps(iommu, &caps);
2330 if (ret)
2331 return ret;
2332
2333 ret = vfio_iommu_info_add_nesting_cap(iommu, &caps);
2334 if (ret)
2335 return ret;
2336
2337 if (caps.size) {
2338 info.flags |= VFIO_IOMMU_INFO_CAPS;
2339
2340 if (info.argsz < sizeof(info) + caps.size) {
2341 info.argsz = sizeof(info) + caps.size;
2342 } else {
2343 vfio_info_cap_shift(&caps, sizeof(info));
2344 if (copy_to_user((void __user *)arg +
2345 sizeof(info), caps.buf,
2346 caps.size)) {
2347 kfree(caps.buf);
2348 return -EFAULT;
2349 }
2350 info.cap_offset = sizeof(info);
2351 }
2352
2353 kfree(caps.buf);
2354 }
2355
2356 return copy_to_user((void __user *)arg, &info, minsz) ?
2357 -EFAULT : 0;
2358
2359 } else if (cmd == VFIO_IOMMU_MAP_DMA) {
2360 struct vfio_iommu_type1_dma_map map;
2361 uint32_t mask = VFIO_DMA_MAP_FLAG_READ |
2362 VFIO_DMA_MAP_FLAG_WRITE;
2363
2364 minsz = offsetofend(struct vfio_iommu_type1_dma_map, size);
2365
2366 if (copy_from_user(&map, (void __user *)arg, minsz))
2367 return -EFAULT;
2368
2369 if (map.argsz < minsz || map.flags & ~mask)
2370 return -EINVAL;
2371
2372 return vfio_dma_do_map(iommu, &map);
2373
2374 } else if (cmd == VFIO_IOMMU_UNMAP_DMA) {
2375 struct vfio_iommu_type1_dma_unmap unmap;
2376 long ret;
2377
2378 minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);
2379
2380 if (copy_from_user(&unmap, (void __user *)arg, minsz))
2381 return -EFAULT;
2382
2383 if (unmap.argsz < minsz || unmap.flags)
2384 return -EINVAL;
2385
2386 ret = vfio_dma_do_unmap(iommu, &unmap);
2387 if (ret)
2388 return ret;
2389
2390 return copy_to_user((void __user *)arg, &unmap, minsz) ?
2391 -EFAULT : 0;
2392
2393 } else if (cmd == VFIO_IOMMU_PASID_REQUEST) {
2394 struct vfio_iommu_type1_pasid_request req;
2395 unsigned long offset;
2396
2397 minsz = offsetofend(struct vfio_iommu_type1_pasid_request,
2398 flags);
2399
2400 if (copy_from_user(&req, (void __user *)arg, minsz))
2401 return -EFAULT;
2402
2403 if (req.argsz < minsz ||
2404 !vfio_iommu_type1_pasid_req_valid(req.flags))
2405 return -EINVAL;
2406
2407 if (copy_from_user((void *)&req + minsz,
2408 (void __user *)arg + minsz,
2409 sizeof(req) - minsz))
2410 return -EFAULT;
2411
2412 switch (req.flags & VFIO_PASID_REQUEST_MASK) {
2413 case VFIO_IOMMU_PASID_ALLOC:
2414 {
2415 int ret = 0, result;
2416
2417 result = vfio_iommu_type1_pasid_alloc(iommu,
2418 req.alloc_pasid.min,
2419 req.alloc_pasid.max);
2420 if (result > 0) {
2421 offset = offsetof(
2422 struct vfio_iommu_type1_pasid_request,
2423 alloc_pasid.result);
2424 ret = copy_to_user(
2425 (void __user *) (arg + offset),
2426 &result, sizeof(result));
2427 } else {
2428 pr_debug("%s: PASID alloc failed\n", __func__);
2429 ret = -EFAULT;
2430 }
2431 return ret;
2432 }
2433 case VFIO_IOMMU_PASID_FREE:
2434 return vfio_iommu_type1_pasid_free(iommu,
2435 req.free_pasid);
2436 default:
2437 return -EINVAL;
2438 }
2439 }
2440
2441 return -ENOTTY;
2442 }
2443
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH 5/8 vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)
drivers/vfio/vfio_iommu_type1.c +56 -0
@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } + + tmp_format = format; + } + ret =0 ; + +out_unlock: +if (format) + *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" ); +return ret; + } header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );
@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; } + nesting_cap->stage1_formats = formats;return 0 ; }
include/uapi/linux/vfio.h +1 -0
@@ -763,6 +763,7 @@ struct vfio_iommu_type1_info_cap_nesting {struct vfio_info_cap_header header;#define VFIO_IOMMU_PASID_REQS (1 << 0)__u32 nesting_capabilities; +__u32 stage1_formats; };#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
kbuild test robot
build botRe: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Thank you for the patch! Yet something to improve:
Hi Yi,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vfio/next]
[also build test ERROR on v5.6-rc6 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Liu-Yi-L/vfio-expose-virtual-Shared-Virtual-Addressing-to-VMs/20200322-213259 base: https://github.com/awilliam/linux-vfio.git next config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
drivers/vfio/vfio_iommu_type1.c: In function 'vfio_iommu_get_stage1_format':
>> drivers/vfio/vfio_iommu_type1.c:2273:4: error: 'DOMAIN_ATTR_PASID_FORMAT' undeclared (first use in this function)
2273 | DOMAIN_ATTR_PASID_FORMAT, &format)) {
| ^~~~~~~~~~~~~~~~~~~~~~~~
drivers/vfio/vfio_iommu_type1.c:2273:4: note: each undeclared identifier is reported only once for each function it appears in
drivers/vfio/vfio_iommu_type1.c: In function 'vfio_iommu_type1_ioctl':
drivers/vfio/vfio_iommu_type1.c:2355:11: error: implicit declaration of function 'iommu_get_uapi_version' [-Werror=implicit-function-declaration]
2355 | return iommu_get_uapi_version();
| ^~~~~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/DOMAIN_ATTR_PASID_FORMAT +2273 drivers/vfio/vfio_iommu_type1.c
2257
2258 static int vfio_iommu_get_stage1_format(struct vfio_iommu *iommu,
2259 u32 *stage1_format)
2260 {
2261 struct vfio_domain *domain;
2262 u32 format = 0, tmp_format = 0;
2263 int ret;
2264
2265 mutex_lock(&iommu->lock);
2266 if (list_empty(&iommu->domain_list)) {
2267 mutex_unlock(&iommu->lock);
2268 return -EINVAL;
2269 }
2270
2271 list_for_each_entry(domain, &iommu->domain_list, next) {
2272 if (iommu_domain_get_attr(domain->domain,
> 2273 DOMAIN_ATTR_PASID_FORMAT, &format)) { 2274 ret = -EINVAL;
2275 format = 0;
2276 goto out_unlock;
2277 }
2278 /*
2279 * format is always non-zero (the first format is
2280 * IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For
2281 * the reason of potential different backed IOMMU
2282 * formats, here we expect to have identical formats
2283 * in the domain list, no mixed formats support.
2284 * return -EINVAL to fail the attempt of setup
2285 * VFIO_TYPE1_NESTING_IOMMU if non-identical formats
2286 * are detected.
2287 */
2288 if (tmp_format && tmp_format != format) {
2289 ret = -EINVAL;
2290 format = 0;
2291 goto out_unlock;
2292 }
2293
2294 tmp_format = format;
2295 }
2296 ret = 0;
2297
2298 out_unlock:
2299 if (format)
2300 *stage1_format = format;
2301 mutex_unlock(&iommu->lock);
2302 return ret;
2303 }
2304
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Liu, Yi LQuoted 73 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)drivers/vfio/vfio_iommu_type1.c +28
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(structvfio_iommu *iommu, return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct · line 2275
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } + + tmp_format = format; + }
this path is invoked only in VFIO_IOMMU_GET_INFO path. If we don't want to assume the status quo that one container holds only one device w/ vIOMMU (the prerequisite for vSVA), looks we also need check the format compatibility when attaching a new group to this container?
Quoted 42 lines
drivers/vfio/vfio_iommu_type1.c +17
-0+ ret =0 ; + +out_unlock: +if (format) + *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" ); +return ret; + } header = vfio_info_cap_add(caps, sizeof(*nesting_cap),
VFIO_IOMMU_TYPE1_INFO_CAP_NESTING,
1);drivers/vfio/vfio_iommu_type1.c +1
-0@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; } + nesting_cap->stage1_formats = formats;drivers/vfio/vfio_iommu_type1.c +0
-0return 0 ; }include/uapi/linux/vfio.h +0
-0
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_nesting { · line 766
Quoted 4 lines
include/uapi/linux/vfio.h +1
-0@@ -763,6 +763,7 @@ struct vfio_iommu_type1_info_cap_nesting {struct vfio_info_cap_header header;#define VFIO_IOMMU_PASID_REQS (1 << 0)__u32 nesting_capabilities; +__u32 stage1_formats;
do you plan to support multiple formats? If not, use singular name.
Quoted 5 lines
include/uapi/linux/vfio.h +0
-1};#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12) --2.7.4
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 86 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 7:49 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 80 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/
stage-1 page table. Therefore, guest format should be compatible with host side.This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 9aa2a67..82a9e0b 100644drivers/vfio/vfio_iommu_type1.c +35
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(structvfio_iommu *iommu, return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + }
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2274
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+ + tmp_format = format; + }this path is invoked only in
VFIO_IOMMU_GET_INFOpath. If we don't want to assume the status quo that one container holds only one device w/ vIOMMU (the prerequisite for vSVA), looks we also need check the format compatibility when attaching a new group to this container?
right. if attaching to a nesting type container (vfio_iommu.nesting bit indicates it), it should check if it is compabile with prior domains in the domain list. But if it is the first one attached to this container, it's fine. is it good?
Quoted 34 lines
Quoted 34 lines
drivers/vfio/vfio_iommu_type1.c +17
-0+ ret =0 ; + +out_unlock: +if (format) + *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" ); +return ret; + } header = vfio_info_cap_add(caps, sizeof(*nesting_cap),
VFIO_IOMMU_TYPE1_INFO_CAP_NESTING,
1);drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2254,6 +2309,7 @@ static intvfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,drivers/vfio/vfio_iommu_type1.c +1
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; } + nesting_cap->stage1_formats = formats;
Quoted 13 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_nesting { · line 766
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +0
-0return 0 ; }include/uapi/linux/vfio.h +1
-0@@ -763,6 +763,7 @@ struct vfio_iommu_type1_info_cap_nesting {struct vfio_info_cap_header header;#define VFIO_IOMMU_PASID_REQS (1 << 0)__u32 nesting_capabilities; +__u32 stage1_formats;do you plan to support multiple formats? If not, use singular name.
I do have such plan. e.g. it may be helpful when one day a platform can support multiple formats.
Regards,
Yi Liu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 97 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 3:38 PMQuoted 42 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 7:49 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 36 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/
stage-1 page table. Therefore, guest format should be compatible with host side.This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int
Quoted 5 lines
(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 9aa2a67..82a9e0b 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static intvfio_iommu_type1_pasid_free(struct
Quoted 36 lines
Quoted 36 lines
drivers/vfio/vfio_iommu_type1.c +32
-0vfio_iommu *iommu, return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ;Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2274
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ format =0 ; + goto out_unlock; + } + + tmp_format = format; + }this path is invoked only in
VFIO_IOMMU_GET_INFOpath. If we don't wantto
Quoted 1 line
assume the status quo that one container holds only one device w/
Quoted 8 lines
vIOMMU
Quoted 2 lines
(the prerequisite for vSVA), looks we also need check the format compatibility when attaching a new group to this container?
right. if attaching to a nesting type container (vfio_iommu.nesting bit indicates it), it should check if it is compabile with prior domains in the domain list. But if it is the first one attached to this container, it's fine. is it good?
yes, but my point is whether we should check the format compatibility in the attach path...
Quoted 56 lines
Quoted 24 lines
Quoted 24 lines
drivers/vfio/vfio_iommu_type1.c +17
-0+ ret =0 ; + +out_unlock: +if (format) + *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" ); +return ret; + }Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +0
-0header = vfio_info_cap_add(caps, sizeof(*nesting_cap),
VFIO_IOMMU_TYPE1_INFO_CAP_NESTING,
1);drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2254,6 +2309,7 @@ static intvfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,
/* nesting iommu type supports PASID requests (alloc/free)*/
Quoted 1 line
Quoted 1 line
nesting_cap->nesting_capabilities|=
VFIO_IOMMU_PASID_REQS;Quoted 16 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +1
-0} + nesting_cap->stage1_formats = formats;return 0 ; }include/uapi/linux/vfio.h +0
-0include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_nesting { · line 766
Quoted 4 lines
include/uapi/linux/vfio.h +1
-0@@ -763,6 +763,7 @@ struct vfio_iommu_type1_info_cap_nesting {struct vfio_info_cap_header header;#define VFIO_IOMMU_PASID_REQS (1 << 0)__u32 nesting_capabilities; +__u32 stage1_formats;do you plan to support multiple formats? If not, use singular name.
I do have such plan. e.g. it may be helpful when one day a platform can support multiple formats.
Regards,
Yi Liu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 109 lines
From: Tian, Kevin <[email protected]>
Sent: Wednesday, April 1, 2020 3:56 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 103 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 3:38 PMQuoted 43 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 7:49 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 37 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported
1st-lvl/stage-1 translation structure format like page table format.Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/
stage-1 page table. Therefore, guest format should be compatible with host side.This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int
Quoted 5 lines
(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 9aa2a67..82a9e0b 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static intvfio_iommu_type1_pasid_free(struct
Quoted 37 lines
Quoted 37 lines
drivers/vfio/vfio_iommu_type1.c +33
-0vfio_iommu *iommu, return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ;Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2274
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ goto out_unlock; + } + + tmp_format = format; + }this path is invoked only in
VFIO_IOMMU_GET_INFOpath. If we don't wantto
Quoted 1 line
assume the status quo that one container holds only one device w/
vIOMMU
Quoted 2 lines
(the prerequisite for vSVA), looks we also need check the format compatibility when attaching a new group to this container?
Quoted 7 lines
Quoted 4 lines
right. if attaching to a nesting type container (vfio_iommu.nesting bit indicates it), it should check if it is compabile with prior domains in the domain list. But if it is the first one attached to this container, it's fine. is it good?
yes, but my point is whether we should check the format compatibility in the attach path...
I guess so. Assume a device has been attached to a container, and userspace has fetched the nesting cap info. e.g. QEMU will have a per-container structure to store the nesting info. And then attach another device from a separate group, if its backend iommu supports different formats, then it will be a problem. If userspace reads the nesting cap info again, it will get a different value. It may affect the prior attched device. If userspace doesn't refresh the nesting info by re-fetch, then the newly added device may use a format which its backend iommu doesn't support.
Although, the vendor specific iommu driver should ensure all devices are backed by iommu units w/ same capability (e.g. format). But it would better to have a check in vfio side all the same. how about your opinion so far?:-)
Regards,
Yi Liu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 132 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 4:07 PMQuoted 77 lines
From: Tian, Kevin <[email protected]>
Sent: Wednesday, April 1, 2020 3:56 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 71 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 3:38 PMQuoted 43 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 7:49 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 37 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported
1st-lvl/stage-1 translation structure format like page table format.Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/
stage-1 page table. Therefore, guest format should be compatible with host side.This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int
Quoted 5 lines
(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 9aa2a67..82a9e0b 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static intvfio_iommu_type1_pasid_free(struct
Quoted 19 lines
Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +15
-0vfio_iommu *iommu, return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { + if (iommu_domain_get_attr(domain->domain,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ DOMAIN_ATTR_PASID_FORMAT, &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats
Quoted 19 lines
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ;Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2274
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ goto out_unlock; + } + + tmp_format = format; + }this path is invoked only in
VFIO_IOMMU_GET_INFOpath. If we don't wantto
Quoted 1 line
assume the status quo that one container holds only one device w/
vIOMMU
Quoted 2 lines
(the prerequisite for vSVA), looks we also need check the format compatibility when attaching a new group to this container?
Quoted 7 lines
Quoted 4 lines
right. if attaching to a nesting type container (vfio_iommu.nesting bit indicates it), it should check if it is compabile with prior domains in the domain list. But if it is the first one attached to this container, it's fine. is it good?
yes, but my point is whether we should check the format compatibility in the attach path...
I guess so. Assume a device has been attached to a container, and userspace has fetched the nesting cap info. e.g. QEMU will have a per-container structure to store the nesting info. And then attach another device from a separate group, if its backend iommu supports different formats, then it will be a problem. If userspace reads the nesting cap info again, it will get a different value. It may affect the prior attched device. If userspace doesn't refresh the nesting info by re-fetch, then the newly added device may use a format which its backend iommu doesn't support.
Quoted 5 lines
Although, the vendor specific iommu driver should ensure all devices are backed by iommu units w/ same capability (e.g. format). But it would better to have a check in vfio side all the same. how about your opinion so far?:-)
I think so.
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 139 lines
From: Tian, Kevin <[email protected]>
Sent: Wednesday, April 1, 2020 4:09 PM
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 134 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 4:07 PMQuoted 78 lines
From: Tian, Kevin <[email protected]>
Sent: Wednesday, April 1, 2020 3:56 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 72 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 3:38 PMQuoted 43 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 7:49 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 37 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported
1st-lvl/stage-1 translation structure format like page table format.Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/
stage-1 page table. Therefore, guest format should be compatible with host side.This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c static int
Quoted 6 lines
(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 9aa2a67..82a9e0b 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static intvfio_iommu_type1_pasid_free(struct
Quoted 19 lines
Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +15
-0vfio_iommu *iommu, return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { + if (iommu_domain_get_attr(domain->domain,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ DOMAIN_ATTR_PASID_FORMAT, &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats
Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ;Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2274
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ goto out_unlock; + } + + tmp_format = format; + }this path is invoked only in
VFIO_IOMMU_GET_INFOpath. If we don't wantto
Quoted 2 lines
assume the status quo that one container holds only one device w/
vIOMMU
Quoted 2 lines
(the prerequisite for vSVA), looks we also need check the format compatibility when attaching a new group to this container?
Quoted 7 lines
Quoted 4 lines
right. if attaching to a nesting type container (vfio_iommu.nesting bit indicates it), it should check if it is compabile with prior domains in the domain list. But if it is the first one attached to this container, it's fine. is it good?
yes, but my point is whether we should check the format compatibility in the attach path...
I guess so. Assume a device has been attached to a container, and userspace has fetched the nesting cap info. e.g. QEMU will have a per-container structure to store the nesting info. And then attach another device from a separate group, if its backend iommu supports different formats, then it will be a problem. If userspace reads the nesting cap info again, it will get a different value. It may affect the prior attched device. If userspace doesn't refresh the nesting info by re-fetch, then the newly added device may use a format which its backend iommu doesn't support.
Quoted 7 lines
Quoted 5 lines
Although, the vendor specific iommu driver should ensure all devices are backed by iommu units w/ same capability (e.g. format). But it would better to have a check in vfio side all the same. how about your opinion so far?:-)
I think so.
Thanks, :-)
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Liu, Yi LHi Yi,
Quoted 36 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)drivers/vfio/vfio_iommu_type1.c +0
-0
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2238
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format)
vfio_pasid_format() to be homogeneous with vfio_pgsize_bitmap() which does the same kind of enumeration of the vfio_iommu domains
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2242
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret;
ret = -EINVAL;
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2245
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+ +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) {
goto out_unlock;
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2252
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) {
I can find DOMAIN_ATTR_PASID_FORMAT in Jacob's v9 but not in v10
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2253
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ ret = -EINVAL ;
could be removed
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2268
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ;
could be removed
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2278
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ format =0 ; + goto out_unlock; + } + + tmp_format = format; + } + ret =0 ; + +out_unlock: +if (format)
if (!ret) ? then you can remove the format = 0 in case of error.
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} + static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2294
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +6
-0struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" );
trace triggered by userspace to be removed?
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2310
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+return ret; + } header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;
What is the meaning for ARM?
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2312
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +1
-0} + nesting_cap->stage1_formats = formats;
as spotted by Kevin, since a single format is supported, rename
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +0
-0return 0 ; }include/uapi/linux/vfio.h +0
-0
include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_nesting { · line 769
Quoted 7 lines
include/uapi/linux/vfio.h +1
-0@@ -763,6 +763,7 @@ struct vfio_iommu_type1_info_cap_nesting {struct vfio_info_cap_header header;#define VFIO_IOMMU_PASID_REQS (1 << 0)__u32 nesting_capabilities; +__u32 stage1_formats; };#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
Thanks
Eric
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Eric,
Quoted 46 lines
From: Auger Eric <[email protected]>
Sent: Wednesday, April 1, 2020 4:51 PM
To: Liu, Yi L <[email protected]>; [email protected]
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceHi Yi,
Quoted 20 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Quoted 8 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56+++++++++++++++++++++++++++++++++++++++++
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct
Quoted 8 lines
include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2238
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +2
-0return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format)
vfio_pasid_format()to be homogeneous withvfio_pgsize_bitmap()which does the same kind of enumeration of thevfio_iommudomains
yes, similar.
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2242
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret;ret =
-EINVAL;
got it.
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2245
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+ +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) {goto
out_unlock;
right.
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2252
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) {I can find
DOMAIN_ATTR_PASID_FORMATin Jacob's v9 but not in v10
oops, I guess he somehow missed. you may find it in below link.
https://github.com/luxis1999/linux-vsva/commit/bf14b11a12f74d58ad3ee626a5d891de395082eb
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct · line 2253
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ ret = -EINVAL ;could be removed
sure.
Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2268
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ;could be removed
got it.
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2278
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ format =0 ; + goto out_unlock; + } + + tmp_format = format; + } + ret =0 ; + +out_unlock: +if (format)if (!ret) ? then you can remove the format = 0 in case of error.
oh, yes.
Quoted 9 lines
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2294
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +6
-0struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" );trace triggered by userspace to be removed?
sure.
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+return ret; + } header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2310
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2312
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +1
-0} + nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
I'll refine the patch per your above comments.
Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Yi,
Quoted 152 lines · Yi L
Hi Eric,
Quoted 46 lines
From: Auger Eric <[email protected]>
Sent: Wednesday, April 1, 2020 4:51 PM
To: Liu, Yi L <[email protected]>; [email protected]
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceHi Yi,
Quoted 20 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Quoted 8 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56+++++++++++++++++++++++++++++++++++++++++
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct
Quoted 8 lines
include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2238
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +2
-0return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format)
vfio_pasid_format()to be homogeneous withvfio_pgsize_bitmap()which does the same kind of enumeration of thevfio_iommudomainsyes, similar.
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2242
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret;ret =
-EINVAL;got it.
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2245
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+ +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) {goto
out_unlock;right.
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2252
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) {I can find
DOMAIN_ATTR_PASID_FORMATin Jacob's v9 but not in v10oops, I guess he somehow missed. you may find it in below link.
https://github.com/luxis1999/linux-vsva/commit/bf14b11a12f74d58ad3ee626a5d891de395082eb
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct · line 2253
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ ret = -EINVAL ;could be removed
sure.
Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2268
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ;could be removed
got it.
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2278
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ format =0 ; + goto out_unlock; + } + + tmp_format = format; + } + ret =0 ; + +out_unlock: +if (format)if (!ret) ? then you can remove the format = 0 in case of error.
oh, yes.
Quoted 9 lines
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+ *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2294
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +6
-0struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" );trace triggered by userspace to be removed?
sure.
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+return ret; + } header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;
Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
Thanks
Eric
Quoted 13 lines
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2312
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +1
-0} + nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
I'll refine the patch per your above comments.
Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 15 lines · Auger Eric
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +0
-0header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.
Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult the IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space. SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be finer grained:
Does the container support:
* VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go with BIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
* VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3
Any bind support implies VFIO_IOMMU_CACHE_INVALIDATE support.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2312
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts of capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Thanks,
Jean
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Jean,
Quoted 45 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2299
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult the IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space.
SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be finer grained:Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3
Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.
good summary. do you expect to see any
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2312
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts of capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g. VTD_STAGE1_FORMAT_V1 which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.
Regards,
Yi Liu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 23 lines
From: Liu, Yi L
Sent: Tuesday, April 7, 2020 5:43 PMQuoted 20 lines
We don't, the PASID spaces are per-VM on Arm, so this function should consult the IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space. SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be finer grained:
Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3
Quoted 3 lines
Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
please ignore this message. I planned to ask if possible to report VFIO_IOMMU_CACHE_INVALIDATE only (no bind support). But I stopped typing it when I came to believe it's unnecessary to report it if there is no bind support.
Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Liu, Yi LHi Yi,
Quoted 63 lines · Yi L
Hi Jean,
Quoted 45 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2299
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ;Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult the IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space.
SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be finer grained:Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2312
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts of capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Quoted 4 lines
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.
But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to VFIO_IOMMU_BIND_GUEST_PGTBL() to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage. My fear is we end up with an overly complex series. This capability getter may be interesting if we can switch to a fallback implementation but here I guess we don't have any fallback. With smmuv3 nested stage we don't have any fallback solution either. For the versions, it is different because the userspace shall be able to adapt (or not) to the max version supported by the kernel.
Thanks
Eric
Quoted 4 lines
Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Quoted 71 lines · Auger Eric
Hi Yi,
Quoted 63 lines · Yi L
Hi Jean,
Quoted 45 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +0
-0header = vfio_info_cap_add(caps, sizeof(*nesting_cap), VFIO_IOMMU_TYPE1_INFO_CAP_NESTING, 1);@@ -2254,6 +2309,7
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 2 lines
(quoted diff) +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |= VFIO_IOMMU_PASID_REQS;Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult the IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space.
SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be finer grained:Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Review
Quoted 2 lines
(quoted diff) +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts of capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Quoted 4 lines
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.
Quoted 4 lines
But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to
VFIO_IOMMU_BIND_GUEST_PGTBL()to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage.
I think we need to present these capabilities at boot time, long before the guest triggers a bind(). For example if the host SMMU doesn't support 16-bit ASID, we need to communicate that to the guest using vSMMU ID registers or PROBE properties. Otherwise a bind() will succeed, but if the guest uses 16-bit ASIDs in its CD, DMA will result in C_BAD_CD events which we'll inject into the guest, for no apparent reason from their perspective.
In addition some VMMs may have fallbacks if shared page tables are not available. They could fall back to a MAP/UNMAP interface, or simply not present a vIOMMU to the guest.
Thanks,
Jean
Quoted 15 lines
My fear is we end up with an overly
complex series. This capability getter may be interesting if we can switch to a fallback implementation but here I guess we don't have any fallback. With smmuv3 nested stage we don't have any fallback solution either. For the versions, it is different because the userspace shall be able to adapt (or not) to the max version supported by the kernel.Thanks
Eric
Quoted 4 lines
Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Jean,
Quoted 77 lines · Jean-Philippe Brucker
Quoted 71 lines · Auger Eric
Hi Yi,
Quoted 63 lines · Yi L
Hi Jean,
Quoted 45 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +0
-0header = vfio_info_cap_add(caps, sizeof(*nesting_cap), VFIO_IOMMU_TYPE1_INFO_CAP_NESTING, 1);@@ -2254,6 +2309,7
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 2 lines
(quoted diff) +0
-0/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |= VFIO_IOMMU_PASID_REQS;Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult the IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space.
SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be finer grained:Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Review
Quoted 2 lines
(quoted diff) +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts of capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Quoted 4 lines
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.Quoted 4 lines
But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to
VFIO_IOMMU_BIND_GUEST_PGTBL()to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage.
Quoted 7 lines
I think we need to present these capabilities at boot time, long before the guest triggers a
bind(). For example if the host SMMU doesn't support 16-bit ASID, we need to communicate that to the guest using vSMMU ID registers or PROBE properties. Otherwise abind()will succeed, but if the guest uses 16-bit ASIDs in its CD, DMA will result inC_BAD_CDevents which we'll inject into the guest, for no apparent reason from their perspective.
OK I understand this case as in this situation we may be able to change the way to iommu is exposed to the guest.
Quoted 4 lines
In addition some VMMs may have fallbacks if shared page tables are not available. They could fall back to a MAP/UNMAP interface, or simply not present a vIOMMU to the guest.
fair enough, there is a need for such capability checker in the mid term. But this patch introduces the capability to check whether system wide PASID alloc is supported and this may not be requested at that stage for the whole vSVM integration?
Thanks
Eric
Quoted 20 lines
Thanks,
JeanQuoted 15 lines
My fear is we end up with an overly
complex series. This capability getter may be interesting if we can switch to a fallback implementation but here I guess we don't have any fallback. With smmuv3 nested stage we don't have any fallback solution either. For the versions, it is different because the userspace shall be able to adapt (or not) to the max version supported by the kernel.Thanks
Eric
Quoted 4 lines
Regards,
Yi Liu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Jean-Philippe BruckerHi Jean,
Quoted 97 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, April 9, 2020 4:15 PM
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 13 lines · Auger Eric
Hi Yi,
Quoted 10 lines · Yi L
Hi Jean,
Quoted 8 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
header = vfio_info_cap_add(caps,
sizeof(*nesting_cap),
VFIO_IOMMU_TYPE1_INFO_CAP_NESTING, 1);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
@@ -2254,6 +2309,7
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 1 line
/* nesting iommu type supports PASID requests (alloc/free)
*/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
nesting_cap->nesting_capabilities|=
VFIO_IOMMU_PASID_REQS;Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult
the
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space.
SMMUv3 support of PASID depends on HW capabilities. So I think this needs tobe
Quoted 20 lines
Quoted 20 lines
Quoted 15 lines
finer grained:
Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts
of
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.Quoted 4 lines
But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to
VFIO_IOMMU_BIND_GUEST_PGTBL()to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage.I think we need to present these capabilities at boot time, long before the guest triggers a
bind(). For example if the host SMMU doesn't support 16-bit ASID, we need to communicate that to the guest using vSMMU ID registers or PROBE properties. Otherwise abind()will succeed, but if the guest uses 16-bit ASIDs in its CD, DMA will result inC_BAD_CDevents which we'll inject into the guest, for no apparent reason from their perspective.
Quoted 4 lines
In addition some VMMs may have fallbacks if shared page tables are not available. They could fall back to a MAP/UNMAP interface, or simply not present a vIOMMU to the guest.
Based on the comments, I think it would be a need to report iommu caps in detail. So I guess iommu uapi needs to provide something alike vfio cap chain in iommu uapi. Please feel free let me know your thoughts. :-)
In vfio, we can define a cap as below:
struct vfio_iommu_type1_info_cap_nesting {
struct vfio_info_cap_header header;
__u64 iommu_model;
#define VFIO_IOMMU_PASID_REQS (1 << 0)
#define VFIO_IOMMU_BIND_GPASID (1 << 1)
#define VFIO_IOMMU_CACHE_INV (1 << 2)
__u32 nesting_capabilities;
__u32 pasid_bits;
#define VFIO_IOMMU_VENDOR_SUB_CAP (1 << 3)
__u32 flags;
__u32 data_size;
__u8 data[]; /*iommu info caps defined by iommu uapi */
};VFIO needs new iommu APIs to ask iommu driver whether PASID/bind_gpasid/ cache_inv/bind_gpasid_table is available or not and also the pasid bits. After that VFIO will ask iommu driver about the iommu_cap_info and fill in the @data[] field.
iommu uapi:
struct iommu_info_cap_header {
__u16 id; /* Identifies capability */
__u16 version; /* Version specific to the capability ID */
__u32 next; /* Offset of next capability */
};#define IOMMU_INFO_CAP_INTEL_VTD 1
struct iommu_info_cap_intel_vtd {
struct iommu_info_cap_header header;
__u32 vaddr_width; /* VA addr_width*/
__u32 ipaddr_width; /* IPA addr_width, input of SL page table */
/* same definition with @flags instruct iommu_gpasid_bind_data_vtd */
__u64 flags;
};#define IOMMU_INFO_CAP_ARM_SMMUv3 2
struct iommu_info_cap_arm_smmuv3 {
struct iommu_info_cap_header header;
...
};Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Yi,
Quoted 99 lines · Yi L
Hi Jean,
Quoted 97 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, April 9, 2020 4:15 PM
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 13 lines · Auger Eric
Hi Yi,
Quoted 10 lines · Yi L
Hi Jean,
Quoted 8 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
header = vfio_info_cap_add(caps,
sizeof(*nesting_cap),
VFIO_IOMMU_TYPE1_INFO_CAP_NESTING, 1);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
@@ -2254,6 +2309,7
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 1 line
/* nesting iommu type supports PASID requests (alloc/free)
*/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
nesting_cap->nesting_capabilities|=
VFIO_IOMMU_PASID_REQS;Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult
the
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space.
SMMUv3 support of PASID depends on HW capabilities. So I think this needs tobe
Quoted 20 lines
Quoted 20 lines
Quoted 15 lines
finer grained:
Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simpler due to
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3Quoted 1 line
Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts
of
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary vendor capability for these?
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.Quoted 4 lines
But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to
VFIO_IOMMU_BIND_GUEST_PGTBL()to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage.I think we need to present these capabilities at boot time, long before the guest triggers a
bind(). For example if the host SMMU doesn't support 16-bit ASID, we need to communicate that to the guest using vSMMU ID registers or PROBE properties. Otherwise abind()will succeed, but if the guest uses 16-bit ASIDs in its CD, DMA will result inC_BAD_CDevents which we'll inject into the guest, for no apparent reason from their perspective.
Quoted 8 lines
Quoted 4 lines
In addition some VMMs may have fallbacks if shared page tables are not available. They could fall back to a MAP/UNMAP interface, or simply not present a vIOMMU to the guest.
Based on the comments, I think it would be a need to report iommu caps in detail. So I guess iommu uapi needs to provide something alike vfio cap chain in iommu uapi. Please feel free let me know your thoughts. :-)
Yes to me it sounds sensible.
Quoted 7 lines
In vfio, we can define a cap as below:
struct vfio_iommu_type1_info_cap_nesting { struct vfio_info_cap_header header; __u64 iommu_model;#define
VFIO_IOMMU_PASID_REQS(1 << 0)
I still think the name shall be changed
Quoted 2 lines
#define
VFIO_IOMMU_BIND_GPASID(1 << 1)
#defineVFIO_IOMMU_CACHE_INV(1 << 2)
this operation seems mandated as soon as we have a nested paging based implementation?
Quoted 29 lines
__u32 nesting_capabilities; __u32 pasid_bits; #define VFIO_IOMMU_VENDOR_SUB_CAP (1 << 3) __u32 flags; __u32 data_size; __u8 data[]; /*iommu info caps defined by iommu uapi */ };VFIO needs new iommu APIs to ask iommu driver whether PASID/bind_gpasid/ cache_inv/bind_gpasid_table is available or not and also the pasid bits. After that VFIO will ask iommu driver about the
iommu_cap_infoand fill in the @data[] field.iommu uapi:
struct iommu_info_cap_header { __u16 id; /* Identifies capability */ __u16 version; /* Version specific to the capability ID */ __u32 next; /* Offset of next capability */ };#define
IOMMU_INFO_CAP_INTEL_VTD1struct iommu_info_cap_intel_vtd { struct iommu_info_cap_header header; __u32 vaddr_width; /* VA addr_width*/ __u32 ipaddr_width; /* IPA addr_width, input of SL page table */ /* same definition with @flags instruct iommu_gpasid_bind_data_vtd */ __u64 flags; };
Quoted 5 lines
#define
IOMMU_INFO_CAP_ARM_SMMUv32struct iommu_info_cap_arm_smmuv3 { struct iommu_info_cap_header header; ... };
Thanks
Eric
Quoted 4 lines
Regards,
Yi Liu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Eric,
Quoted 124 lines
From: Auger Eric <[email protected]>
Sent: Friday, April 10, 2020 11:28 AM
To: Liu, Yi L <[email protected]>; Jean-Philippe Brucker <jean-
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceHi Yi,
Quoted 25 lines · Yi L
Hi Jean,
Quoted 23 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, April 9, 2020 4:15 PM
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 12 lines · Auger Eric
Hi Yi,
Quoted 9 lines · Yi L
Hi Jean,
Quoted 7 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, April 3, 2020 4:23 PM
To: Auger Eric <[email protected]> userspaceQuoted 2 lines · Auger Eric
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
header = vfio_info_cap_add(caps,
sizeof(*nesting_cap),
VFIO_IOMMU_TYPE1_INFO_CAP_NESTING, 1);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
@@ -2254,6 +2309,7
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
@@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 1 line
/* nesting iommu type supports PASID requests
(alloc/free)
Quoted 26 lines
Quoted 26 lines
*/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
nesting_cap->nesting_capabilities|=
VFIO_IOMMU_PASID_REQS;Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
What is the meaning for ARM?
I think it's just a software capability exposed to userspace, on userspace side, it has a choice to use it or not. :-) The reason define it and report it in cap nesting is that I'd like to make the pasid alloc/free be available just for IOMMU with type
VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good for ARM. We can find a proper way to report the availability.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Well it is more a question for jean-Philippe. Do we have a system wide PASID allocation on ARM?
We don't, the PASID spaces are per-VM on Arm, so this function should consult
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
IOMMU driver before setting flags. As you said on patch 3, nested doesn't necessarily imply PASID support. The SMMUv2 does not support PASID but does support nesting stages 1 and 2 for the IOVA space. SMMUv3 support of PASID depends on HW capabilities. So I think this needs to
be
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
finer grained:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Does the container support:
*VFIO_IOMMU_PASID_REQUEST?
-> Yes for VT-d 3
-> No for Arm SMMU
* VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL?
-> Yes for VT-d 3
-> Sometimes for SMMUv2
-> No for SMMUv3 (if we go withBIND_PASID_TABLE, which is simplerdue to
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
PASID tables being in GPA space.)
*VFIO_IOMMU_BIND_PASID_TABLE?
-> No for VT-d
-> Sometimes for SMMUv3Any bind support implies
VFIO_IOMMU_CACHE_INVALIDATEsupport.good summary. do you expect to see any
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Review
Quoted 2 lines
(quoted diff) +1
-0+ nesting_cap->stage1_formats = formats;as spotted by Kevin, since a single format is supported, rename
ok, I was believing it may be possible on ARM or so. :-) will rename it.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I guess we could have a secondary
vendor capability for these?
Quoted 24 lines
Quoted 19 lines
Quoted 10 lines
Quoted 5 lines
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1st level page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to
VFIO_IOMMU_BIND_GUEST_PGTBL()to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage.I think we need to present these capabilities at boot time, long before the guest triggers a
bind(). For example if the host SMMU doesn't support 16-bit ASID, we need to communicate that to the guest using vSMMU ID registers or PROBE properties. Otherwise abind()will succeed, but if the guest uses 16-bit ASIDs in its CD, DMA will result inC_BAD_CDevents which we'll inject into the guest, for no apparent reason from their perspective.Quoted 4 lines
In addition some VMMs may have fallbacks if shared page tables are not available. They could fall back to a MAP/UNMAP interface, or simply not present a vIOMMU to the guest.
Quoted 4 lines
Based on the comments, I think it would be a need to report iommu caps in detail. So I guess iommu uapi needs to provide something alike vfio cap chain in iommu uapi. Please feel free let me know your thoughts. :-)
Yes to me it sounds sensible.
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
In vfio, we can define a cap as below:
struct vfio_iommu_type1_info_cap_nesting { struct vfio_info_cap_header header; __u64 iommu_model;#define
VFIO_IOMMU_PASID_REQS(1 << 0)I still think the name shall be changed
yes, I'll rename it per your suggestion.:-)
Quoted 4 lines
Quoted 2 lines
#define
VFIO_IOMMU_BIND_GPASID(1 << 1)
#defineVFIO_IOMMU_CACHE_INV(1 << 2)this operation seems mandated as soon as we have a nested paging based implementation?
oh, yes, should be. will remove it and comment in the code.
Regards,
Yi Liu
Quoted 35 lines
Quoted 29 lines
__u32 nesting_capabilities; __u32 pasid_bits; #define VFIO_IOMMU_VENDOR_SUB_CAP (1 << 3) __u32 flags; __u32 data_size; __u8 data[]; /*iommu info caps defined by iommu uapi */ };VFIO needs new iommu APIs to ask iommu driver whether PASID/bind_gpasid/ cache_inv/bind_gpasid_table is available or not and also the pasid bits. After that VFIO will ask iommu driver about the
iommu_cap_infoand fill in the @data[] field.iommu uapi:
struct iommu_info_cap_header { __u16 id; /* Identifies capability */ __u16 version; /* Version specific to the capability ID */ __u32 next; /* Offset of next capability */ };#define
IOMMU_INFO_CAP_INTEL_VTD1struct iommu_info_cap_intel_vtd { struct iommu_info_cap_header header; __u32 vaddr_width; /* VA addr_width*/ __u32 ipaddr_width; /* IPA addr_width, input of SL page table */ /* same definition with @flags instruct iommu_gpasid_bind_data_vtd */ __u64 flags; };Quoted 5 lines
#define
IOMMU_INFO_CAP_ARM_SMMUv32struct iommu_info_cap_arm_smmuv3 { struct iommu_info_cap_header header; ... };
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Regards,
Yi Liu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Liu, Yi LHi Jean, Eric,
Quoted 5 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, April 9, 2020 8:47 PM
Subject: RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
[...]
Quoted 29 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
capabilities for page and PASID tables (granules, GPA size, ASID/PASID size,
HW
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean
much. I
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
guess we could have a secondary vendor capability for these?
Actually, I'm wondering if we can define some formats to stands for a set of capabilities. e.g.
VTD_STAGE1_FORMAT_V1which may indicates the 1stlevel
Quoted 16 lines
Quoted 7 lines
Quoted 2 lines
page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse the capabilities.
But eventually do we really need all those capability getters? I mean can't we simply rely on the actual call to
VFIO_IOMMU_BIND_GUEST_PGTBL()to detect any mismatch? Definitively the error handling may be heavier on userspace but can't we manage.I think we need to present these capabilities at boot time, long before the guest triggers a
bind(). For example if the host SMMU doesn't support 16-bit ASID, we need to communicate that to the guest using vSMMU ID registers or PROBE properties. Otherwise abind()will succeed, but if the guest uses 16-bit ASIDs in its CD, DMA will result inC_BAD_CDevents which we'll inject into the guest, for no apparent reason from their perspective.
Quoted 8 lines
Quoted 4 lines
In addition some VMMs may have fallbacks if shared page tables are not available. They could fall back to a MAP/UNMAP interface, or simply not present a vIOMMU to the guest.
Based on the comments, I think it would be a need to report iommu caps in detail. So I guess iommu uapi needs to provide something alike vfio cap chain in iommu uapi. Please feel free let me know your thoughts. :-)
Consider more, I guess it may be better to start simpler. Cap chain suits the case in which there are multiple caps. e.g. some vendor iommu driver may want to report iommu capabilities via multiple caps. Actually, in VT-d side, the host IOMMU capability could be reported in a single cap structure. I'm not sure about ARM side. Will there be multiple iommu_info_caps for ARM?
Quoted 16 lines
In vfio, we can define a cap as below:
struct vfio_iommu_type1_info_cap_nesting { struct vfio_info_cap_header header; __u64 iommu_model; #define VFIO_IOMMU_PASID_REQS (1 << 0) #define VFIO_IOMMU_BIND_GPASID (1 << 1) #define VFIO_IOMMU_CACHE_INV (1 << 2) __u32 nesting_capabilities; __u32 pasid_bits; #define VFIO_IOMMU_VENDOR_SUB_CAP (1 << 3) __u32 flags; __u32 data_size; __u8 data[]; /*iommu info caps defined by iommu uapi */ };
If iommu vendor driver only needs one cap structure to report hw capability, then I think we needn't implement cap chain in iommu uapi. The @data[] field could be determined by the @iommu_model and @flags fields. This would be easier. thoughts?
Quoted 21 lines
VFIO needs new iommu APIs to ask iommu driver whether PASID/bind_gpasid/ cache_inv/bind_gpasid_table is available or not and also the pasid bits. After that VFIO will ask iommu driver about the
iommu_cap_infoand fill in the @data[] field.iommu uapi:
struct iommu_info_cap_header { __u16 id; /* Identifies capability */ __u16 version; /* Version specific to the capability ID */ __u32 next; /* Offset of next capability */ };#define
IOMMU_INFO_CAP_INTEL_VTD1struct iommu_info_cap_intel_vtd { struct iommu_info_cap_header header; __u32 vaddr_width; /* VA addr_width*/ __u32 ipaddr_width; /* IPA addr_width, input of SL page table */ /* same definition with @flags instruct iommu_gpasid_bind_data_vtd */ __u64 flags; };
Quoted 5 lines
#define
IOMMU_INFO_CAP_ARM_SMMUv32struct iommu_info_cap_arm_smmuv3 { struct iommu_info_cap_header header; ... };
Regards,
Yi Liu
Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
In reply to Liu, Yi LQuoted 88 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)drivers/vfio/vfio_iommu_type1.c +47
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } + + tmp_format = format; + } + ret =0 ; + +out_unlock: +if (format) + *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps)
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2295
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +7
-0{struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; + + ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" ); +return ret;
Looks like this generates a warning and causes the iommu_get_info ioctl to fail if the hardware doesn't support the pasid format attribute, or the domain list is empty. This breaks users on existing hardware.
Quoted 24 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ } header =vfio_info_cap_add (caps,sizeof (*nesting_cap),VFIO_IOMMU_TYPE1_INFO_CAP_NESTING ,1 );@@ -2254,6 +2309,7 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,/* nesting iommu type supports PASID requests (alloc/free) */ nesting_cap->nesting_capabilities |=VFIO_IOMMU_PASID_REQS ; } + nesting_cap->stage1_formats = formats;return 0 ; }include/uapi/linux/vfio.h +1
-0@@ -763,6 +763,7 @@ struct vfio_iommu_type1_info_cap_nesting {struct vfio_info_cap_header header;#define VFIO_IOMMU_PASID_REQS (1 << 0)__u32 nesting_capabilities; +__u32 stage1_formats; };#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
RE: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace
Hi Alex,
Quoted 105 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 3:20 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspaceQuoted 20 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO exposes IOMMU nesting translation (a.k.a dual stage translation) capability to userspace. Thus applications like QEMU could support vIOMMU with hardware's nesting translation capability for pass-through devices. Before setting up nesting translation for pass-through devices, QEMU and other applications need to learn the supported 1st-lvl/stage-1 translation structure format like page table format.
Take vSVA (virtual Shared Virtual Addressing) as an example, to support vSVA for pass-through devices, QEMU setup nesting translation for pass- through devices. The guest page table are configured to host as 1st-lvl/ stage-1 page table. Therefore, guest format should be compatible with host side.
This patch reports the supported 1st-lvl/stage-1 page table format on the current platform to userspace. QEMU and other alike applications should use this format info when trying to setup IOMMU nesting translation on host IOMMU.
Quoted 8 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 56+++++++++++++++++++++++++++++++++++++++++
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct
Quoted 8 lines
include/uapi/linux/vfio.h | 1 + 2 files changed, 57 insertions(+)drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2234,11 +2234,66 @@ static int vfio_iommu_type1_pasid_free(struct
vfio_iommu*iommu,Quoted 58 lines
drivers/vfio/vfio_iommu_type1.c +50
-0return ret; } +static int vfio_iommu_get_stage1_format (struct vfio_iommu *iommu, +u32 *stage1_format) +{ +struct vfio_domain *domain; +u32 format =0 , tmp_format =0 ; +int ret; + +mutex_lock (&iommu->lock); +if (list_empty (&iommu->domain_list)) { +mutex_unlock (&iommu->lock); +return -EINVAL ; + } + +list_for_each_entry (domain, &iommu->domain_list, next) { +if (iommu_domain_get_attr (domain->domain, +DOMAIN_ATTR_PASID_FORMAT , &format)) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } +/* +* format is always non-zero (the first format is +* IOMMU_PASID_FORMAT_INTEL_VTD which is 1). For +* the reason of potential different backed IOMMU +* formats, here we expect to have identical formats +* in the domain list, no mixed formats support. +* return -EINVAL to fail the attempt of setup +* VFIO_TYPE1_NESTING_IOMMU if non-identical formats +* are detected. +*/ +if (tmp_format && tmp_format != format) { + ret = -EINVAL ; + format =0 ; + goto out_unlock; + } + + tmp_format = format; + } + ret =0 ; + +out_unlock: +if (format) + *stage1_format = format; +mutex_unlock (&iommu->lock); +return ret; +} +static int vfio_iommu_info_add_nesting_cap (struct vfio_iommu *iommu,struct vfio_info_cap *caps) {struct vfio_info_cap_header *header;struct vfio_iommu_type1_info_cap_nesting *nesting_cap; +u32 formats =0 ; +int ret; +
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_type1_pasid_free(struct vfio_iommu *iommu, · line 2295
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ ret =vfio_iommu_get_stage1_format (iommu, &formats); +if (ret) { +pr_warn ("Failed to get stage-1 format\n" ); +return ret;Looks like this generates a warning and causes the
iommu_get_infoioctl to fail if the hardware doesn't support the pasid format attribute, or the domain list is empty. This breaks users on existing hardware.
oops, yes, it should not fail anything as it is just an extended feature. let me correct it.
Thanks,
Yi Liu
PATCH 6/8 vfio/type1: Bind guest page tables to host
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMU is an IOMMU type which is backed by hardware IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first- level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.
This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.
In this patch, guest page table bind and unbind are done by using flags VFIO_IOMMU_BIND_GUEST_PGTBL and VFIO_IOMMU_UNBIND_GUEST_PGTBL under IOCTL VFIO_IOMMU_BIND, the bind/unbind data are conveyed by
struct iommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host via VFIO_IOMMU_PASID_REQUEST.
Bind guest translation structures (here is guest page table) to host are the first step to setup vSVA (Virtual Shared Virtual Addressing).
--- drivers/vfio/vfio_iommu_type1.c | 158 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)
drivers/vfio/vfio_iommu_type1.c +158 -0
@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/*
@@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,return 0 ; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +} + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); +} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); +} + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. +*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to +* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/ +vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {
@@ -2471,6 +2580,55 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) { +struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; + +/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ; + + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + } + +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;
include/uapi/linux/vfio.h +46 -0
58 lines not on this page · raw message
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
kbuild test robot
build botRe: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Thank you for the patch! Yet something to improve:
Hi Yi,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vfio/next]
[also build test ERROR on v5.6-rc6 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Liu-Yi-L/vfio-expose-virtual-Shared-Virtual-Addressing-to-VMs/20200322-213259 base: https://github.com/awilliam/linux-vfio.git next config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
drivers/vfio/vfio_iommu_type1.c: In function 'vfio_iommu_get_stage1_format':
drivers/vfio/vfio_iommu_type1.c:2300:4: error: 'DOMAIN_ATTR_PASID_FORMAT' undeclared (first use in this function)
2300 | DOMAIN_ATTR_PASID_FORMAT, &format)) {
| ^~~~~~~~~~~~~~~~~~~~~~~~
drivers/vfio/vfio_iommu_type1.c:2300:4: note: each undeclared identifier is reported only once for each function it appears in
drivers/vfio/vfio_iommu_type1.c: In function 'vfio_iommu_type1_ioctl':
drivers/vfio/vfio_iommu_type1.c:2464:11: error: implicit declaration of function 'iommu_get_uapi_version' [-Werror=implicit-function-declaration]
2464 | return iommu_get_uapi_version();
| ^~~~~~~~~~~~~~~~~~~~~~
>> drivers/vfio/vfio_iommu_type1.c:2626:15: error: implicit declaration of function 'iommu_uapi_get_data_size' [-Werror=implicit-function-declaration]
2626 | data_size = iommu_uapi_get_data_size(
| ^~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/vfio/vfio_iommu_type1.c:2627:5: error: 'IOMMU_UAPI_BIND_GPASID' undeclared (first use in this function)
2627 | IOMMU_UAPI_BIND_GPASID, version);
| ^~~~~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/iommu_uapi_get_data_size +2626 drivers/vfio/vfio_iommu_type1.c
2446
2447 static long vfio_iommu_type1_ioctl(void *iommu_data,
2448 unsigned int cmd, unsigned long arg)
2449 {
2450 struct vfio_iommu *iommu = iommu_data;
2451 unsigned long minsz;
2452
2453 if (cmd == VFIO_CHECK_EXTENSION) {
2454 switch (arg) {
2455 case VFIO_TYPE1_IOMMU:
2456 case VFIO_TYPE1v2_IOMMU:
2457 case VFIO_TYPE1_NESTING_IOMMU:
2458 return 1;
2459 case VFIO_DMA_CC_IOMMU:
2460 if (!iommu)
2461 return 0;
2462 return vfio_domains_have_iommu_cache(iommu);
2463 case VFIO_NESTING_IOMMU_UAPI:
2464 return iommu_get_uapi_version();
2465 default:
2466 return 0;
2467 }
2468 } else if (cmd == VFIO_IOMMU_GET_INFO) {
2469 struct vfio_iommu_type1_info info;
2470 struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
2471 unsigned long capsz;
2472 int ret;
2473
2474 minsz = offsetofend(struct vfio_iommu_type1_info, iova_pgsizes);
2475
2476 /* For backward compatibility, cannot require this */
2477 capsz = offsetofend(struct vfio_iommu_type1_info, cap_offset);
2478
2479 if (copy_from_user(&info, (void __user *)arg, minsz))
2480 return -EFAULT;
2481
2482 if (info.argsz < minsz)
2483 return -EINVAL;
2484
2485 if (info.argsz >= capsz) {
2486 minsz = capsz;
2487 info.cap_offset = 0; /* output, no-recopy necessary */
2488 }
2489
2490 info.flags = VFIO_IOMMU_INFO_PGSIZES;
2491
2492 info.iova_pgsizes = vfio_pgsize_bitmap(iommu);
2493
2494 ret = vfio_iommu_iova_build_caps(iommu, &caps);
2495 if (ret)
2496 return ret;
2497
2498 ret = vfio_iommu_info_add_nesting_cap(iommu, &caps);
2499 if (ret)
2500 return ret;
2501
2502 if (caps.size) {
2503 info.flags |= VFIO_IOMMU_INFO_CAPS;
2504
2505 if (info.argsz < sizeof(info) + caps.size) {
2506 info.argsz = sizeof(info) + caps.size;
2507 } else {
2508 vfio_info_cap_shift(&caps, sizeof(info));
2509 if (copy_to_user((void __user *)arg +
2510 sizeof(info), caps.buf,
2511 caps.size)) {
2512 kfree(caps.buf);
2513 return -EFAULT;
2514 }
2515 info.cap_offset = sizeof(info);
2516 }
2517
2518 kfree(caps.buf);
2519 }
2520
2521 return copy_to_user((void __user *)arg, &info, minsz) ?
2522 -EFAULT : 0;
2523
2524 } else if (cmd == VFIO_IOMMU_MAP_DMA) {
2525 struct vfio_iommu_type1_dma_map map;
2526 uint32_t mask = VFIO_DMA_MAP_FLAG_READ |
2527 VFIO_DMA_MAP_FLAG_WRITE;
2528
2529 minsz = offsetofend(struct vfio_iommu_type1_dma_map, size);
2530
2531 if (copy_from_user(&map, (void __user *)arg, minsz))
2532 return -EFAULT;
2533
2534 if (map.argsz < minsz || map.flags & ~mask)
2535 return -EINVAL;
2536
2537 return vfio_dma_do_map(iommu, &map);
2538
2539 } else if (cmd == VFIO_IOMMU_UNMAP_DMA) {
2540 struct vfio_iommu_type1_dma_unmap unmap;
2541 long ret;
2542
2543 minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);
2544
2545 if (copy_from_user(&unmap, (void __user *)arg, minsz))
2546 return -EFAULT;
2547
2548 if (unmap.argsz < minsz || unmap.flags)
2549 return -EINVAL;
2550
2551 ret = vfio_dma_do_unmap(iommu, &unmap);
2552 if (ret)
2553 return ret;
2554
2555 return copy_to_user((void __user *)arg, &unmap, minsz) ?
2556 -EFAULT : 0;
2557
2558 } else if (cmd == VFIO_IOMMU_PASID_REQUEST) {
2559 struct vfio_iommu_type1_pasid_request req;
2560 unsigned long offset;
2561
2562 minsz = offsetofend(struct vfio_iommu_type1_pasid_request,
2563 flags);
2564
2565 if (copy_from_user(&req, (void __user *)arg, minsz))
2566 return -EFAULT;
2567
2568 if (req.argsz < minsz ||
2569 !vfio_iommu_type1_pasid_req_valid(req.flags))
2570 return -EINVAL;
2571
2572 if (copy_from_user((void *)&req + minsz,
2573 (void __user *)arg + minsz,
2574 sizeof(req) - minsz))
2575 return -EFAULT;
2576
2577 switch (req.flags & VFIO_PASID_REQUEST_MASK) {
2578 case VFIO_IOMMU_PASID_ALLOC:
2579 {
2580 int ret = 0, result;
2581
2582 result = vfio_iommu_type1_pasid_alloc(iommu,
2583 req.alloc_pasid.min,
2584 req.alloc_pasid.max);
2585 if (result > 0) {
2586 offset = offsetof(
2587 struct vfio_iommu_type1_pasid_request,
2588 alloc_pasid.result);
2589 ret = copy_to_user(
2590 (void __user *) (arg + offset),
2591 &result, sizeof(result));
2592 } else {
2593 pr_debug("%s: PASID alloc failed\n", __func__);
2594 ret = -EFAULT;
2595 }
2596 return ret;
2597 }
2598 case VFIO_IOMMU_PASID_FREE:
2599 return vfio_iommu_type1_pasid_free(iommu,
2600 req.free_pasid);
2601 default:
2602 return -EINVAL;
2603 }
2604
2605 } else if (cmd == VFIO_IOMMU_BIND) {
2606 struct vfio_iommu_type1_bind bind;
2607 u32 version;
2608 int data_size;
2609 void *gbind_data;
2610 int ret;
2611
2612 minsz = offsetofend(struct vfio_iommu_type1_bind, flags);
2613
2614 if (copy_from_user(&bind, (void __user *)arg, minsz))
2615 return -EFAULT;
2616
2617 if (bind.argsz < minsz)
2618 return -EINVAL;
2619
2620 /* Get the version of struct iommu_gpasid_bind_data */
2621 if (copy_from_user(&version,
2622 (void __user *) (arg + minsz),
2623 sizeof(version)))
2624 return -EFAULT;
2625
> 2626 data_size = iommu_uapi_get_data_size( > 2627 IOMMU_UAPI_BIND_GPASID, version);
2628 gbind_data = kzalloc(data_size, GFP_KERNEL);
2629 if (!gbind_data)
2630 return -ENOMEM;
2631
2632 if (copy_from_user(gbind_data,
2633 (void __user *) (arg + minsz), data_size)) {
2634 kfree(gbind_data);
2635 return -EFAULT;
2636 }
2637
2638 switch (bind.flags & VFIO_IOMMU_BIND_MASK) {
2639 case VFIO_IOMMU_BIND_GUEST_PGTBL:
2640 ret = vfio_iommu_type1_bind_gpasid(iommu,
2641 gbind_data);
2642 break;
2643 case VFIO_IOMMU_UNBIND_GUEST_PGTBL:
2644 ret = vfio_iommu_type1_unbind_gpasid(iommu,
2645 gbind_data);
2646 break;
2647 default:
2648 ret = -EINVAL;
2649 break;
2650 }
2651 kfree(gbind_data);
2652 return ret;
2653 }
2654
2655 return -ENOTTY;
2656 }
2657
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
In reply to Liu, Yi LQuoted 32 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by hardware
IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first- level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind
guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLandVFIO_IOMMU_UNBIND_GUEST_PGTBLunder IOCTLVFIO_IOMMU_BIND, the bind/unbind data are conveyed by
structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.
Quoted 1 line
Bind guest translation structures (here is guest page table) to host
Bind -> Binding
Quoted 1 line
are the first step to setup vSVA (Virtual Shared Virtual Addressing).
are -> is. and you already explained vSVA earlier.
Quoted 55 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 158 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/*
drivers/vfio/vfio_iommu_type1.c static int · line 2350
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +6
-0@@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, return 0; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; +
In Jacob's vSVA iommu series, [PATCH 06/11]:
+ /* REVISIT: upper layer/VFIO can track host process that bind the PASID.
+ * ioasid_set = mm might be sufficient for vfio to check pasid VMM
+ * ownership.
+ */
I asked him who exactly should be responsible for tracking the pasid ownership. Although no response yet, I expect vfio/iommu can have a clear policy and also documented here to provide consistent message.
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2360
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +11
-0+return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +} + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid);
curious why we have to share the same bind_data structure between bind and unbind, especially when unbind requires only one field? I didn't see a clear reason, and just similar to earlier ALLOC/FREE which don't share structure either. Current way simply wastes space for unbind operation...
Quoted 27 lines
drivers/vfio/vfio_iommu_type1.c +26
-0+} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); +} + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data);
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2396
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. +*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to
binds -> bindings
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2400
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/
remove 'switch'
Quoted 29 lines
drivers/vfio/vfio_iommu_type1.c +25
-0+vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + static long vfio_iommu_type1_ioctl(void *iommu_data,drivers/vfio/vfio_iommu_type1.c +0
-0unsigned int cmd, unsigned long arg) {
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void · line 2585
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +2
-0@@ -2471,6 +2580,55 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, default:return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) {
BIND what? VFIO_IOMMU_BIND_PASID sounds clearer to me.
Quoted 90 lines
drivers/vfio/vfio_iommu_type1.c +47
-0+struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; + +/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ; + + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + } + +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }drivers/vfio/vfio_iommu_type1.c +0
-0return -ENOTTY ;include/uapi/linux/vfio.h +4
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for
+ * nesting type IOMMUs. In @data field It takes struct
+ *iommu_gpasid_bind_data.
+ * -VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page
table operation
+ * invoked byVFIO_IOMMU_BIND_GUEST_PGTBL.
+ *
+ */include/uapi/linux/vfio.h +9
-0+struct vfio_iommu_type1_bind { +__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0) +#define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1) +__u8 data[]; +}; + +#define VFIO_IOMMU_BIND_MASK (VFIO_IOMMU_BIND_GUEST_PGTBL| \
+
VFIO_IOMMU_UNBIND_GUEST_PGTBL)
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 883
Quoted 8 lines
include/uapi/linux/vfio.h +8
-0+ +/** +* VFIO_IOMMU_BIND - _IOW(VFIO_TYPE, VFIO_BASE + 23, +* struct vfio_iommu_type1_bind) +* +* Manage address spaces of devices in this container. Initially a TYPE1 +* container can only have one address space, managed with +* VFIO_IOMMU_MAP/UNMAP_DMA.
the last sentence seems irrelevant and more suitable in commit msg.
Quoted 6 lines
+ *
+ * An IOMMU of typeVFIO_TYPE1_NESTING_IOMMUcan be managed by both MAP/UNMAP
+ * and BIND ioctls at the same time. MAP/UNMAP acts on the stage-2 (host) page
+ * tables, and BIND manages the stage-1 (guest) page tables. Other types of
Are "other types" the counterpart to VFIO_TYPE1_NESTING_IOMMU? What are those types? I thought only NESTING_IOMMU allows two stage translation...
Quoted 1 line
+ * IOMMU may allow MAP/UNMAP and BIND to coexist, where
The first sentence said the same thing. Then what is the exact difference?
Quoted 7 lines
MAP/UNMAP controls
+ * the traffics only require single stage translation while BIND controls the
+ * traffics require nesting translation. But this depends on the underlying
+ * IOMMU architecture and isn't guaranteed. Example of this is the guest SVA
+ * traffics, such traffics need nesting translation to gaingVA->gPAand then
+ *gPA->hPAtranslation.
I'm a bit confused about the content since "other types of". Are they trying to state some exceptions/corner cases that this API cannot resolve or explain the desired behavior of the API? Especially the last example, which is worded as if the example for "isn't guaranteed" but isn't guest SVA the main purpose of this API?
Quoted 13 lines
+ *
+ * Availability of this feature depends on the device, its bus, the underlying
+ * IOMMU and the CPU architecture.
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +2
-1+#define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /* -- 2.7.4
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 31 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 8:46 PM
Subject: RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 24 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by hardware IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first-
level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.Quoted 2 lines
In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLand
VFIO_IOMMU_UNBIND_GUEST_PGTBL
Quoted 8 lines
Quoted 6 lines
under IOCTL
VFIO_IOMMU_BIND, the bind/unbind data are conveyed by structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.Bind guest translation structures (here is guest page table) to host
Bind -> Binding
got it.
Quoted 3 lines
Quoted 1 line
are the first step to setup vSVA (Virtual Shared Virtual Addressing).
are -> is. and you already explained vSVA earlier.
oh yes, it is.
Quoted 72 lines
Quoted 54 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 158 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 82a9e0b..a877747 100644drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* drivers/vfio/vfio_iommu_type1.c static int · line 2349
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +5
-0@@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, return 0; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; +In Jacob's vSVA iommu series, [PATCH 06/11]:
(quoted diff) +1
-0+ /* REVISIT: upper layer/VFIO can track host process that bind thePASID.
(quoted diff) +3
-0+ * ioasid_set = mm might be sufficient for vfio to check pasid VMM + * ownership. + */
Quoted 3 lines
I asked him who exactly should be responsible for tracking the pasid ownership. Although no response yet, I expect vfio/iommu can have a clear policy and also documented here to provide consistent message.
yep.
Quoted 6 lines
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); } + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data;
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2360
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+ +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid);curious why we have to share the same
bind_datastructure between bind and unbind, especially when unbind requires only one field? I didn't see a clear reason, and just similar to earlier ALLOC/FREE which don't share structure either. Current way simply wastes space for unbind operation...
no special reason today. But the gIOVA support over nested translation is in plan, it may require a flag to indicate it as guest iommu driver may user a single PASID value(RID2PASID) for all devices in guest side. Especially if the RID2PASID value used for IOVA the the same with host side. So adding a flag to indicate the binding is for IOVA is helpful. For PF/VF, iommu driver just bind with the host side's RID2PASID. While for ADI (Assignable Device Interface), vfio layer needs to figure out the default PASID stored in the aux-domain, and then iommu driver bind gIOVA table to the default PASID. The potential flag is required in both bind and unbind path. As such, it would be better to share the structure.
Quoted 28 lines
Quoted 28 lines
drivers/vfio/vfio_iommu_type1.c +27
-0+} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) { +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); } + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) { +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2396
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. +*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to binds -> bindings
got it.
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2400
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/ remove 'switch'
oh, yes.
Quoted 28 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) { +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + }drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2471,6 +2580,55 @@
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2584
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +2
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.
Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's called VFIO_IOMMU_BIND.
Quoted 94 lines
Quoted 94 lines
drivers/vfio/vfio_iommu_type1.c +47
-0+struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; + +/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ; + + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + } + +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +4
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for
+ * nesting type IOMMUs. In @data field It takes struct
+ *iommu_gpasid_bind_data.
+ * -VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page
table operation
+ * invoked byVFIO_IOMMU_BIND_GUEST_PGTBL.
+ *
+ */include/uapi/linux/vfio.h +9
-0+struct vfio_iommu_type1_bind { +__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0) +#define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1) +__u8 data[]; +}; + +#define VFIO_IOMMU_BIND_MASK (VFIO_IOMMU_BIND_GUEST_PGTBL| \
+
VFIO_IOMMU_UNBIND_GUEST_PGTBL)include/uapi/linux/vfio.h +3
-0+ +/** +* VFIO_IOMMU_BIND - _IOW(VFIO_TYPE, VFIO_BASE + 23,
Quoted 8 lines
Quoted 6 lines
+ * struct
vfio_iommu_type1_bind)
+ *
+ * Manage address spaces of devices in this container. Initially a +TYPE1
+ * container can only have one address space, managed with
+ * VFIO_IOMMU_MAP/UNMAP_DMA.the last sentence seems irrelevant and more suitable in commit msg.
oh, I could remove it.
Quoted 9 lines
Quoted 8 lines
+ *
+ * An IOMMU of typeVFIO_TYPE1_NESTING_IOMMUcan be managed by both MAP/UNMAP
+ * and BIND ioctls at the same time. MAP/UNMAP acts on the stage-2
+ (host)
page
+ * tables, and BIND manages the stage-1 (guest) page tables. Other
+ types of
Quoted 3 lines
Are "other types" the counterpart to
VFIO_TYPE1_NESTING_IOMMU? What are those types? I thought onlyNESTING_IOMMUallows two stage translation...
it's a mistake... please ignore this message. would correct it in next version.
Quoted 4 lines
Quoted 2 lines
+ * IOMMU may allow MAP/UNMAP and BIND to coexist, where
The first sentence said the same thing. Then what is the exact difference?
this sentence were added by mistake. will correct it.
Quoted 13 lines
Quoted 12 lines
MAP/UNMAP controls
+ * the traffics only require single stage translation while BIND
+ controls the
+ * traffics require nesting translation. But this depends on the
+ underlying
+ * IOMMU architecture and isn't guaranteed. Example of this is the
+ guest
SVA
+ * traffics, such traffics need nesting translation to gaingVA->gPA
+ and then
+ *gPA->hPAtranslation.
Quoted 6 lines
I'm a bit confused about the content since "other types of". Are they trying to state some exceptions/corner cases that this API cannot resolve or explain the desired behavior of the API? Especially the last example, which is worded as if the example for "isn't guaranteed"
but isn't guest SVA the main purpose of this API?
I think the description in original patch is bad especially with the "other types" phrase. How about the below description?
/**
* VFIO_IOMMU_BIND - _IOW(VFIO_TYPE, VFIO_BASE + 23,
* struct vfio_iommu_type1_bind)
*
* Manage address spaces of devices in this container when it's an IOMMU
* of type VFIO_TYPE1_NESTING_IOMMU. Such type IOMMU allows MAP/UNMAP and
* BIND to coexist, where MAP/UNMAP controls the traffics only require
* single stage translation while BIND controls the traffics require nesting
* translation.
*
* Availability of this feature depends on the device, its bus, the underlying
* IOMMU and the CPU architecture.
*
* returns: 0 on success, -errno on failure.
*/
Regards,
Yi Liu
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 147 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 5:13 PMQuoted 10 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 8:46 PM
Subject: RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 6 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed byhardware
Quoted 21 lines
Quoted 18 lines
IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first-
level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.Quoted 2 lines
In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLand
VFIO_IOMMU_UNBIND_GUEST_PGTBLQuoted 8 lines
Quoted 6 lines
under IOCTL
VFIO_IOMMU_BIND, the bind/unbind data are conveyed by structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.Bind guest translation structures (here is guest page table) to host
Bind -> Binding
got it.
Quoted 3 lines
Quoted 1 line
are the first step to setup vSVA (Virtual Shared Virtual Addressing).
are -> is. and you already explained vSVA earlier.
oh yes, it is.
Quoted 66 lines
Quoted 54 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 158 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 82a9e0b..a877747 100644drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* drivers/vfio/vfio_iommu_type1.c static int · line 2349
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +5
-0@@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, return 0; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; +In Jacob's vSVA iommu series, [PATCH 06/11]:
Review
Quoted 1 line
(quoted diff) +1
-0+ /* REVISIT: upper layer/VFIO can track host process that bindthe
Review
Quoted 2 lines
PASID.
(quoted diff) +1
-0+ * ioasid_set = mm might be sufficient for vfio to check pasidVMM
Quoted 4 lines
(quoted diff) +2
-0+ * ownership. + */I asked him who exactly should be responsible for tracking the pasid
ownership.
Quoted 1 line
Although no response yet, I expect vfio/iommu can have a clear policy and
also
Quoted 1 line
documented here to provide consistent message.
yep.
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2360
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); } + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid);curious why we have to share the same
bind_datastructure between bindand
Quoted 1 line
unbind, especially when unbind requires only one field? I didn't see a clear
reason,
Quoted 2 lines
and just similar to earlier ALLOC/FREE which don't share structure either. Current way simply wastes space for unbind operation...
Quoted 10 lines
no special reason today. But the gIOVA support over nested translation is in plan, it may require a flag to indicate it as guest iommu driver may user a single PASID value(RID2PASID) for all devices in guest side. Especially if the RID2PASID value used for IOVA the the same with host side. So adding a flag to indicate the binding is for IOVA is helpful. For PF/VF, iommu driver just bind with the host side's RID2PASID. While for ADI (Assignable Device Interface), vfio layer needs to figure out the default PASID stored in the aux-domain, and then iommu driver bind gIOVA table to the default PASID. The potential flag is required in both bind and unbind path. As such, it would be better to share the structure.
I'm fine with it if you are pretty sure that more extension will be required in the future, though I didn't fully understand above explanations. 😊
Quoted 89 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2368
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data){
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2375
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); } + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data){
Quoted 14 lines
Quoted 14 lines
drivers/vfio/vfio_iommu_type1.c +14
-0+int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2396
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. +*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to binds -> bindings
got it.
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2400
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/ remove 'switch'
oh, yes.
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu*iommu,
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c static int · line 2405
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ struct iommu_gpasid_bind_data *gbind_data){
Quoted 9 lines
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@-
Quoted 9 lines
2471,6 +2580,55 @@
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2584
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +2
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.
Quoted 7 lines
Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.
but aren't they all about PASID related binding?
Quoted 173 lines
Quoted 81 lines
Quoted 81 lines
drivers/vfio/vfio_iommu_type1.c +47
-0+struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; + +/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ; + + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + } + +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +4
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for
+ * nesting type IOMMUs. In @data field It takes struct
+ *iommu_gpasid_bind_data.
+ * -VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page
table operation
+ * invoked byVFIO_IOMMU_BIND_GUEST_PGTBL.
+ *
+ */
+structvfio_iommu_type1_bind{Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request {
Quoted 8 lines
include/uapi/linux/vfio.h +8
-0+__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0) +#define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1) +__u8 data[]; +}; + +#define VFIO_IOMMU_BIND_MASK (
VFIO_IOMMU_BIND_GUEST_PGTBLQuoted 6 lines
Quoted 6 lines
| \
+
VFIO_IOMMU_UNBIND_GUEST_PGTBL)include/uapi/linux/vfio.h +3
-0+ +/** +* VFIO_IOMMU_BIND - _IOW(VFIO_TYPE, VFIO_BASE + 23, Quoted 8 lines
Quoted 6 lines
+ * struct
vfio_iommu_type1_bind)
+ *
+ * Manage address spaces of devices in this container. Initially a +TYPE1
+ * container can only have one address space, managed with
+ * VFIO_IOMMU_MAP/UNMAP_DMA.the last sentence seems irrelevant and more suitable in commit msg.
oh, I could remove it.
Quoted 2 lines
Quoted 2 lines
+ *
+ * An IOMMU of typeVFIO_TYPE1_NESTING_IOMMUcan be managedby
Quoted 10 lines
Quoted 6 lines
both MAP/UNMAP
+ * and BIND ioctls at the same time. MAP/UNMAP acts on the stage-2
+ (host)
page
+ * tables, and BIND manages the stage-1 (guest) page tables. Other
+ types ofAre "other types" the counterpart to
VFIO_TYPE1_NESTING_IOMMU? What are those types? I thought onlyNESTING_IOMMUallows two stage translation...it's a mistake... please ignore this message. would correct it in next version.
Quoted 4 lines
Quoted 2 lines
+ * IOMMU may allow MAP/UNMAP and BIND to coexist, where
The first sentence said the same thing. Then what is the exact difference?
this sentence were added by mistake. will correct it.
Quoted 13 lines
Quoted 12 lines
MAP/UNMAP controls
+ * the traffics only require single stage translation while BIND
+ controls the
+ * traffics require nesting translation. But this depends on the
+ underlying
+ * IOMMU architecture and isn't guaranteed. Example of this is the
+ guest
SVA
+ * traffics, such traffics need nesting translation to gaingVA->gPA
+ and then
+ *gPA->hPAtranslation.Quoted 1 line
I'm a bit confused about the content since "other types of". Are they trying
to state
Quoted 1 line
some exceptions/corner cases that this API cannot resolve or explain the
desired
Quoted 1 line
behavior of the API? Especially the last example, which is worded as if the
example
Quoted 3 lines
for "isn't guaranteed"
but isn't guest SVA the main purpose of this API?I think the description in original patch is bad especially with the "other types"
phrase. How about the below description?/**
*VFIO_IOMMU_BIND- _IOW(VFIO_TYPE,VFIO_BASE+ 23,
* structvfio_iommu_type1_bind)
*
* Manage address spaces of devices in this container when it's an IOMMU
* of typeVFIO_TYPE1_NESTING_IOMMU. Such type IOMMU allows
MAP/UNMAP and
* BIND to coexist, where MAP/UNMAP controls the traffics only require
* single stage translation while BIND controls the traffics require nesting
* translation.
*
* Availability of this feature depends on the device, its bus, the underlying
* IOMMU and the CPU architecture.
*
* returns: 0 on success, -errno on failure.
*/
Quoted 2 lines
Regards,
Yi Liu
yes, this looks better.
Thanks
Kevin
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 270 lines
From: Tian, Kevin <[email protected]>
Sent: Thursday, April 2, 2020 10:12 AM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 153 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, April 1, 2020 5:13 PMQuoted 11 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 8:46 PM
Subject: RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 6 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed byhardware
Quoted 27 lines
Quoted 19 lines
IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the
first-
level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.Quoted 2 lines
In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLand
VFIO_IOMMU_UNBIND_GUEST_PGTBLQuoted 5 lines
under IOCTL
VFIO_IOMMU_BIND, the bind/unbind data are conveyed by structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.Quoted 4 lines
Quoted 2 lines
Bind guest translation structures (here is guest page table) to host
Bind -> Binding
got it.
Quoted 3 lines
Quoted 1 line
are the first step to setup vSVA (Virtual Shared Virtual Addressing).
are -> is. and you already explained vSVA earlier.
oh yes, it is.
Quoted 66 lines
Quoted 54 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 158 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 82a9e0b..a877747 100644drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* drivers/vfio/vfio_iommu_type1.c static int · line 2349
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +5
-0@@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, return 0; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; +In Jacob's vSVA iommu series, [PATCH 06/11]:
Review
Quoted 1 line
(quoted diff) +1
-0+ /* REVISIT: upper layer/VFIO can track host process that bindthe
Review
Quoted 2 lines
PASID.
(quoted diff) +1
-0+ * ioasid_set = mm might be sufficient for vfio to check pasidVMM
Quoted 4 lines
(quoted diff) +2
-0+ * ownership. + */I asked him who exactly should be responsible for tracking the pasid
ownership.
Quoted 2 lines
Although no response yet, I expect vfio/iommu can have a clear policy and
also
Quoted 1 line
documented here to provide consistent message.
yep.
Quoted 4 lines
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); } + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data;Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2360
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid);curious why we have to share the same
bind_datastructure between bindand
Quoted 2 lines
unbind, especially when unbind requires only one field? I didn't see a clear
reason,
Quoted 2 lines
and just similar to earlier ALLOC/FREE which don't share structure either. Current way simply wastes space for unbind operation...
Quoted 10 lines
no special reason today. But the gIOVA support over nested translation is in plan, it may require a flag to indicate it as guest iommu driver may user a single PASID value(RID2PASID) for all devices in guest side. Especially if the RID2PASID value used for IOVA the the same with host side. So adding a flag to indicate the binding is for IOVA is helpful. For PF/VF, iommu driver just bind with the host side's RID2PASID. While for ADI (Assignable Device Interface), vfio layer needs to figure out the default PASID stored in the aux-domain, and then iommu driver bind gIOVA table to the default PASID. The potential flag is required in both bind and unbind path. As such, it would be better to share the
structure.
I'm fine with it if you are pretty sure that more extension will be required in the future, though I didn't fully understand above explanations. 😊
Quoted 89 lines
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2368
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data){
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2375
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); } + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data){
Quoted 14 lines
Quoted 14 lines
drivers/vfio/vfio_iommu_type1.c +14
-0+int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2396
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. +*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to binds -> bindings
got it.
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2400
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/ remove 'switch'
oh, yes.
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+vfio_iommu_type1_do_guest_unbind (iommu, gbind_data); + +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu*iommu,
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c static int · line 2405
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ struct iommu_gpasid_bind_data *gbind_data){
Quoted 9 lines
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@-
Quoted 9 lines
2471,6 +2580,55 @@
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2584
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +2
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.Quoted 7 lines
Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.
Quoted 1 line
but aren't they all about PASID related binding?
yeah, I can rename it. :-)
Regards,
Yi Liu
Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 17 lines · Yi L
Quoted 17 lines
Quoted 16 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2584
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +2
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.
Quoted 3 lines
Quoted 1 line
but aren't they all about PASID related binding?
yeah, I can rename it. :-)
I don't know if anyone intends to implement it, but SMMUv2 supports nesting translation without any PASID support. For that case the name VFIO_IOMMU_BIND_GUEST_PGTBL without "PASID" anywhere makes more sense. Ideally we'd also use a neutral name for the IOMMU API instead of bind_gpasid(), but that's easier to change later.
Thanks,
Jean
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Hi Jean,
Quoted 24 lines
From: Jean-Philippe Brucker < [email protected] >
Sent: Friday, April 3, 2020 4:35 PM
Subject: Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 19 lines · Yi L
Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2584
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +2
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.Quoted 7 lines
Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.but aren't they all about PASID related binding?
Quoted 7 lines
Quoted 1 line
yeah, I can rename it. :-)
I don't know if anyone intends to implement it, but SMMUv2 supports nesting translation without any PASID support. For that case the name
VFIO_IOMMU_BIND_GUEST_PGTBLwithout "PASID" anywhere makes more sense. Ideally we'd also use a neutral name for the IOMMU API instead ofbind_gpasid(), but that's easier to change later.
I agree VFIO_IOMMU_BIND is somehow not straight-forward. Especially, it may cause confusion when thinking about VFIO_SET_IOMMU. How about using VFIO_NESTING_IOMMU_BIND_STAGE1 to cover a) and b)? And has another VFIO_BIND_PROCESS in future for the SVA bind case.
Regards,
Yi Liu
Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 34 lines · Yi L
Hi Jean,
Quoted 24 lines
From: Jean-Philippe Brucker < [email protected] >
Sent: Friday, April 3, 2020 4:35 PM
Subject: Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 19 lines · Yi L
Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Review
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,(quoted diff) +2
-0default: return -EINVAL; } + + } else if (cmd == VFIO_IOMMU_BIND) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.Quoted 7 lines
Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.but aren't they all about PASID related binding?
Quoted 7 lines
Quoted 1 line
yeah, I can rename it. :-)
I don't know if anyone intends to implement it, but SMMUv2 supports nesting translation without any PASID support. For that case the name
VFIO_IOMMU_BIND_GUEST_PGTBLwithout "PASID" anywhere makes more sense. Ideally we'd also use a neutral name for the IOMMU API instead ofbind_gpasid(), but that's easier to change later.
Quoted 4 lines
I agree
VFIO_IOMMU_BINDis somehow not straight-forward. Especially, it may cause confusion when thinking aboutVFIO_SET_IOMMU. How about usingVFIO_NESTING_IOMMU_BIND_STAGE1to cover a) and b)? And has anotherVFIO_BIND_PROCESSin future for the SVA bind case.
I think minimizing the number of ioctls is more important than finding the ideal name. VFIO_IOMMU_BIND was fine to me, but if it's too vague then rename it to VFIO_IOMMU_BIND_PASID and we'll just piggy-back on it for non-PASID things (they should be rare enough).
Thanks,
Jean
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 45 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, April 9, 2020 4:29 PM
To: Liu, Yi L <[email protected]>Quoted 31 lines · Yi L
Hi Jean,
Quoted 29 lines
From: Jean-Philippe Brucker < [email protected] >
Sent: Friday, April 3, 2020 4:35 PM
Subject: Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 17 lines · Yi L
Quoted 17 lines
Quoted 16 lines
Quoted 8 lines
Review
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,(quoted diff) +2
-0default: return -EINVAL; } + + } else if (cmd == VFIO_IOMMU_BIND) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.Quoted 3 lines
Quoted 1 line
but aren't they all about PASID related binding?
yeah, I can rename it. :-)
I don't know if anyone intends to implement it, but SMMUv2 supports nesting translation without any PASID support. For that case the name
VFIO_IOMMU_BIND_GUEST_PGTBLwithout "PASID" anywhere makes moresense.
Quoted 7 lines
Quoted 2 lines
Ideally we'd also use a neutral name for the IOMMU API instead of
bind_gpasid(), but that's easier to change later.I agree
VFIO_IOMMU_BINDis somehow not straight-forward. Especially, it may cause confusion when thinking aboutVFIO_SET_IOMMU. How about usingVFIO_NESTING_IOMMU_BIND_STAGE1to cover a) and b)? And has anotherVFIO_BIND_PROCESSin future for the SVA bind case.
Quoted 4 lines
I think minimizing the number of ioctls is more important than finding the ideal name.
VFIO_IOMMU_BINDwas fine to me, but if it's too vague then rename it toVFIO_IOMMU_BIND_PASIDand we'll just piggy-back on it for non-PASID things (they should be rare enough).
maybe we can start with VFIO_IOMMU_BIND_PASID. Actually, there is also a discussion on reusing the same ioctl and vfio structure for pasid_alloc/free, bind/unbind_gpasid. and cache_inv. how about your opinion?
https://lkml.org/lkml/2020/4/3/833
Regards,
Yi Liu
Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 45 lines · Yi L
Quoted 45 lines
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, April 9, 2020 4:29 PM
To: Liu, Yi L <[email protected]>Quoted 31 lines · Yi L
Hi Jean,
Quoted 29 lines
From: Jean-Philippe Brucker < [email protected] >
Sent: Friday, April 3, 2020 4:35 PM
Subject: Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 17 lines · Yi L
Quoted 17 lines
Quoted 16 lines
Quoted 8 lines
Review
Quoted 6 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,(quoted diff) +2
-0default: return -EINVAL; } + + } else if (cmd == VFIO_IOMMU_BIND) {BIND what?
VFIO_IOMMU_BIND_PASIDsounds clearer to me.Emm, it's up to the flags to indicate bind what. It was proposed to cover the three cases below:
a) BIND/UNBIND_GPASID
b) BIND/UNBIND_GPASID_TABLE
c) BIND/UNBIND_PROCESS
<only a) is covered in this patch>
So it's calledVFIO_IOMMU_BIND.Quoted 3 lines
Quoted 1 line
but aren't they all about PASID related binding?
yeah, I can rename it. :-)
I don't know if anyone intends to implement it, but SMMUv2 supports nesting translation without any PASID support. For that case the name
VFIO_IOMMU_BIND_GUEST_PGTBLwithout "PASID" anywhere makes moresense.
Quoted 7 lines
Quoted 2 lines
Ideally we'd also use a neutral name for the IOMMU API instead of
bind_gpasid(), but that's easier to change later.I agree
VFIO_IOMMU_BINDis somehow not straight-forward. Especially, it may cause confusion when thinking aboutVFIO_SET_IOMMU. How about usingVFIO_NESTING_IOMMU_BIND_STAGE1to cover a) and b)? And has anotherVFIO_BIND_PROCESSin future for the SVA bind case.
Quoted 8 lines
Quoted 4 lines
I think minimizing the number of ioctls is more important than finding the ideal name.
VFIO_IOMMU_BINDwas fine to me, but if it's too vague then rename it toVFIO_IOMMU_BIND_PASIDand we'll just piggy-back on it for non-PASID things (they should be rare enough).maybe we can start with
VFIO_IOMMU_BIND_PASID. Actually, there is also a discussion on reusing the same ioctl and vfio structure for pasid_alloc/free, bind/unbind_gpasid. andcache_inv. how about your opinion?
Merging bind with unbind and alloc with free makes sense. I'd leave at least invalidate a separate ioctl, because alloc/bind/unbind/free are setup functions while invalidate is a runtime thing and performance sensitive. But I can't see a good reason not to merge them all together, so either way is fine by me.
Thanks,
Jean
Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
In reply to Liu, Yi LQuoted 122 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by hardware IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first- level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLandVFIO_IOMMU_UNBIND_GUEST_PGTBLunder IOCTLVFIO_IOMMU_BIND, the bind/unbind data are conveyed by
structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.Bind guest translation structures (here is guest page table) to host are the first step to setup vSVA (Virtual Shared Virtual Addressing).
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 158 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)drivers/vfio/vfio_iommu_type1.c +66
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* @@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,return 0 ; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +} + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); +} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); +} + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock;
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2392
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group.
Where was the non-singleton group restriction done, I missed that.
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2401
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to +* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/ +vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);
However, the for_each_dev function stops when the callback function returns error, are we just assuming we stop at the same device as we faulted on the first time and that we traverse the same set of devices the second time? It seems strange to me that unbind should be able to fail.
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data)
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2419
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);
How is a user supposed to respond to their unbind failing?
Quoted 24 lines
drivers/vfio/vfio_iommu_type1.c +17
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2471,6 +2580,55 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) { +struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz))
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2603
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; + +/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ;
Why are we coping things from beyond the size we've validated that the user has provided again?
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2615
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + }
And again. argsz isn't just for minsz.
Quoted 32 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+ +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +1
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 863
Quoted 10 lines
include/uapi/linux/vfio.h +7
-0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for +* nesting type IOMMUs. In @data field It takes struct +* iommu_gpasid_bind_data. +* - VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page table operation +* invoked by VFIO_IOMMU_BIND_GUEST_PGTBL.
This must require iommu_gpasid_bind_data in the data field as well, right?
Quoted 41 lines
include/uapi/linux/vfio.h +38
-0+* +*/ +struct vfio_iommu_type1_bind { +__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_BIND_GUEST_PGTBL (1 << 0) +#define VFIO_IOMMU_UNBIND_GUEST_PGTBL (1 << 1) +__u8 data[]; +}; + +#define VFIO_IOMMU_BIND_MASK (VFIO_IOMMU_BIND_GUEST_PGTBL | \ + VFIO_IOMMU_UNBIND_GUEST_PGTBL) + +/** +* VFIO_IOMMU_BIND - _IOW(VFIO_TYPE, VFIO_BASE + 23, +* struct vfio_iommu_type1_bind) +* +* Manage address spaces of devices in this container. Initially a TYPE1 +* container can only have one address space, managed with +* VFIO_IOMMU_MAP/UNMAP_DMA. +* +* An IOMMU of type VFIO_TYPE1_NESTING_IOMMU can be managed by both MAP/UNMAP +* and BIND ioctls at the same time. MAP/UNMAP acts on the stage-2 (host) page +* tables, and BIND manages the stage-1 (guest) page tables. Other types of +* IOMMU may allow MAP/UNMAP and BIND to coexist, where MAP/UNMAP controls +* the traffics only require single stage translation while BIND controls the +* traffics require nesting translation. But this depends on the underlying +* IOMMU architecture and isn't guaranteed. Example of this is the guest SVA +* traffics, such traffics need nesting translation to gain gVA->gPA and then +* gPA->hPA translation. +* +* Availability of this feature depends on the device, its bus, the underlying +* IOMMU and the CPU architecture. +* +* returns: 0 on success, -errno on failure. +*/ +#define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*