vfio: expose virtual Shared Virtual Addressing to VMs
Subsystems (2)
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
In reply to Alex WilliamsonHi Alex,
Quoted 136 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 3:57 AM
To: Liu, Yi L <[email protected]>Quoted 19 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by hardware IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first- level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.Quoted 2 lines
In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLandVFIO_IOMMU_UNBIND_GUEST_PGTBLunder IOCTL
Quoted 15 lines
VFIO_IOMMU_BIND, the bind/unbind data are conveyed by
structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.Bind guest translation structures (here is guest page table) to host are the first step to setup vSVA (Virtual Shared Virtual Addressing).
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio_iommu_type1.c | 158++++++++++++++++++++++++++++++++++++++++
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct
Quoted 42 lines
include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* @@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 46 lines
drivers/vfio/vfio_iommu_type1.c +43
-0return 0 ; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +} + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); +} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); +} + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data);
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2392
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. Where was the non-singleton group restriction done, I missed that.
Hmm, it's missed. thanks for spotting it. How about adding this check in the vfio_iommu_for_each_dev()? If looped a non-singleton group, just skip it. It applies to the cache_inv path all the same.
Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to +* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch.
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct · line 2401
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+*/ +vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);However, the
for_each_devfunction stops when the callback function returns error, are we just assuming we stop at the same device as we faulted on the first time and that we traverse the same set of devices the second time? It seems strange to me that unbind should be able to fail.
unbind can fail if a user attempts to unbind a pasid which is not belonged to it or a pasid which hasn't ever been bound. Otherwise, I didn't see a reason to fail.
Quoted 12 lines
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock);
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2419
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);How is a user supposed to respond to their unbind failing?
If it's a malicious unbind (e.g. unbind a not yet bound pasid or unbind a pasid which doesn't belong to current user).
Quoted 30 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2471,6 +2580,55 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +16
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) { +struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; +
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2603
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ;Why are we coping things from beyond the size we've validated that the user has provided again?
let me wait for the result in Jacob's thread below. looks like need to have a decision from you and Joreg. If using argsze is good, then I guess we don't need the version-to-size mapping. right? Actually, the version-to-size mapping is added to ensure vfio copy data correctly.
https://lkml.org/lkml/2020/4/2/876
Quoted 57 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2615
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + }And again. argsz isn't just for minsz.
Quoted 32 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+ +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +1
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 862
Quoted 9 lines
include/uapi/linux/vfio.h +6
-0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for +* nesting type IOMMUs. In @data field It takes struct +* iommu_gpasid_bind_data. +* - VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page table operation
Quoted 4 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 863
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* invoked by VFIO_IOMMU_BIND_GUEST_PGTBL. This must require
iommu_gpasid_bind_datain the data field as well, right?
yes.
Regards,
Yi Liu
Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Quoted 147 lines · Yi L
Hi Alex,
Quoted 136 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 3:57 AM
To: Liu, Yi L <[email protected]>Quoted 19 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by hardware IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first- level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs. This would be introduced in next patch.Quoted 2 lines
In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLandVFIO_IOMMU_UNBIND_GUEST_PGTBLunder IOCTL
Quoted 15 lines
VFIO_IOMMU_BIND, the bind/unbind data are conveyed by
structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host viaVFIO_IOMMU_PASID_REQUEST.Bind guest translation structures (here is guest page table) to host are the first step to setup vSVA (Virtual Shared Virtual Addressing).
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio_iommu_type1.c | 158++++++++++++++++++++++++++++++++++++++++
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct
Quoted 42 lines
include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* @@ -2314,6 +2341,88 @@ static int vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 46 lines
drivers/vfio/vfio_iommu_type1.c +43
-0return 0 ; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +} + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); +} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); +} + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2392
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. Where was the non-singleton group restriction done, I missed that.
Quoted 4 lines
Hmm, it's missed. thanks for spotting it. How about adding this check in the
vfio_iommu_for_each_dev()? If looped a non-singleton group, just skip it. It applies to thecache_invpath all the same.
I don't really understand the singleton issue, which is why I was surprised to see this since I didn't see a discussion previously. Skipping a singleton group seems like unpredictable behavior to the user though.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to +* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct · line 2401
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+*/ +vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);However, the
for_each_devfunction stops when the callback function returns error, are we just assuming we stop at the same device as we faulted on the first time and that we traverse the same set of devices the second time? It seems strange to me that unbind should be able to fail.
Quoted 3 lines
unbind can fail if a user attempts to unbind a pasid which is not belonged to it or a pasid which hasn't ever been bound. Otherwise, I didn't see a reason to fail.
Even if so, this doesn't address the first part of the question. If our for_each_dev() callback returns error then the loop stops and we can't be sure we've triggered it everywhere that it needs to be triggered. There are also aspects of whether it's an error to unbind something that is not bound because the result is still that the pasid is unbound, right?
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2419
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);How is a user supposed to respond to their unbind failing?
Quoted 2 lines
If it's a malicious unbind (e.g. unbind a not yet bound pasid or unbind a pasid which doesn't belong to current user).
And if it's not a malicious unbind? To me this is similar semantics to free() failing. Is there any remedy other than to abort? Thanks,
Alex
Quoted 112 lines
Quoted 30 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2471,6 +2580,55 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +16
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) { +struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; +Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2603
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ;Why are we coping things from beyond the size we've validated that the user has provided again?
let me wait for the result in Jacob's thread below. looks like need to have a decision from you and Joreg. If using argsze is good, then I guess we don't need the version-to-size mapping. right? Actually, the version-to-size mapping is added to ensure vfio copy data correctly.
https://lkml.org/lkml/2020/4/2/876Quoted 57 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2615
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + }And again. argsz isn't just for minsz.
Quoted 32 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+ +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +1
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 862
Quoted 9 lines
include/uapi/linux/vfio.h +6
-0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for +* nesting type IOMMUs. In @data field It takes struct +* iommu_gpasid_bind_data. +* - VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page table operation
Quoted 4 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 863
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* invoked by VFIO_IOMMU_BIND_GUEST_PGTBL. This must require
iommu_gpasid_bind_datain the data field as well, right?yes.
Regards,
Yi Liu
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
Hi Alex,
Quoted 155 lines
From: Alex Williamson <[email protected]>
Sent: Saturday, April 4, 2020 2:11 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 28 lines · Yi L
Hi Alex,
Quoted 26 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 3:57 AM
To: Liu, Yi L <[email protected]>Quoted 19 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by hardware IOMMUs that have nesting DMA translation (a.k.a dual stage address translation). For such hardware IOMMUs, there are two stages/levels of address translation, and software may let userspace/VM to own the first-
level/stage-1 translation structures. Example of such usage is vSVA ( virtual Shared Virtual Addressing). VM owns the first-level/stage-1 translation structures and bind the structures to host, then hardware IOMMU would utilize nesting translation when doing DMA translation fo the devices behind such hardware IOMMU.This patch adds vfio support for binding guest translation (a.k.a stage 1) structure to host iommu. And for
VFIO_TYPE1_NESTING_IOMMU, not only bind guest page table is
needed, it also requires to expose interface to guest for iommu cache invalidation when guest modified the first-level/stage-1 translation structures since hardware needs to be notified to flush stale iotlbs.This would be introduced in next patch.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
In this patch, guest page table bind and unbind are done by using flags
VFIO_IOMMU_BIND_GUEST_PGTBLandVFIO_IOMMU_UNBIND_GUEST_PGTBLunder IOCTL
Quoted 3 lines
VFIO_IOMMU_BIND, the bind/unbind data are conveyed by structiommu_gpasid_bind_data. Before binding guest page table to host, VM should have got a PASID allocated by host via
VFIO_IOMMU_PASID_REQUEST.Quoted 109 lines
Quoted 108 lines
Quoted 12 lines
Bind guest translation structures (here is guest page table) to host are the first step to setup vSVA (Virtual Shared Virtual Addressing).
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio_iommu_type1.c | 158++++++++++++++++++++++++++++++++++++++++
Quoted 42 lines
include/uapi/linux/vfio.h | 46 ++++++++++++ 2 files changed, 204 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 82a9e0b..a877747 100644drivers/vfio/vfio_iommu_type1.c +27
-0@@ -130,6 +130,33 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +struct domain_capsule { +struct iommu_domain *domain; +void *data; +}; + +/* iommu->lock must be held */ +static int vfio_iommu_for_each_dev (struct vfio_iommu *iommu, +int (*fn)(struct device *dev,void *data), +void *data) +{ +struct domain_capsule dc = {.data = data}; +struct vfio_domain *d; +struct vfio_group *g; +int ret =0 ; + +list_for_each_entry (d, &iommu->domain_list, next) { + dc.domain = d->domain; +list_for_each_entry (g, &d->group_list, next) { + ret =iommu_group_for_each_dev (g->iommu_group, + &dc, fn); +if (ret) +break ; + } + } +return ret; +} +static int put_pfn (unsigned long pfn,int prot);/* @@ -2314,6 +2341,88 @@ static intQuoted 1 line
vfio_iommu_info_add_nesting_cap(struct
vfio_iommu*iommu,Quoted 37 lines
drivers/vfio/vfio_iommu_type1.c +34
-0return 0 ; } +static int vfio_bind_gpasid_fn (struct device *dev,void *data) { +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); } + +static int vfio_unbind_gpasid_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_gpasid_bind_data *gbind_data = + (struct iommu_gpasid_bind_data *) dc->data; + +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); +} + +/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) { +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); } + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) { +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock;drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2392
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. Where was the non-singleton group restriction done, I missed that.
Quoted 7 lines
Quoted 3 lines
Hmm, it's missed. thanks for spotting it. How about adding this check in the
vfio_iommu_for_each_dev()? If looped a non-singleton group, just skip it. It applies to thecache_invpath all the same.I don't really understand the singleton issue, which is why I was surprised to see this since I didn't see a discussion previously.
Skipping a singleton group seems like unpredictable behavior to the user though.
There is a discussion on the SVA availability in the below link. There was a conclusion to only support SVA for singleton group. I think bind guest page table also needs to apply this rule.
https://patchwork.kernel.org/patch/10213877/
Quoted 20 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to +* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+*/ +vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);However, the
for_each_devfunction stops when the callback function returns error, are we just assuming we stop at the same device as we faulted on the first time and that we traverse the same set of devices the second time? It seems strange to me that unbind should be able to fail.Quoted 3 lines
unbind can fail if a user attempts to unbind a pasid which is not belonged to it or a pasid which hasn't ever been bound. Otherwise, I didn't see a reason to fail.
Quoted 5 lines
Even if so, this doesn't address the first part of the question. If our
for_each_dev()
callback returns error then the loop stops and we can't be sure we've triggered it
everywhere that it needs to be triggered.
Hmm, let me pull back a little. Back to the code, in the attempt to do bind, the code uses for_each_dev() to loop devices. If failed then uses for_each_dev() again to do unbind. Your question is can the second for_each_dev() be able to undo the bind correctly as the second for_each_dev() call has no idea where it failed in the bind phase. is it? Actually, this is why I added the comment that no need to check the return value of vfio_iommu_type1_do_guest_unbind().
Quoted 4 lines
There are also aspects of whether it's an
error to unbind something that is not bound because the result is still that the pasid
is unbound, right?
agreed, as you mentioned in the below comment, no need to fail unbind unless user is trying to unbind a pasid which doesn't belong to it.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} + +static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu,drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2419
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ struct iommu_gpasid_bind_data *gbind_data) { +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);How is a user supposed to respond to their unbind failing?
Quoted 5 lines
Quoted 2 lines
If it's a malicious unbind (e.g. unbind a not yet bound pasid or unbind a pasid which doesn't belong to current user).
And if it's not a malicious unbind? To me this is similar semantics to
free()failing. Is there any remedy other than to abort? Thanks,
got it. so if user is trying to unbind a pasid which doesn't belong to it, should kernel return error to user or just abort it?
Regards,
Yi Liu
Quoted 116 lines
Alex
Quoted 8 lines
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +6
-0+ +out_unlock: +mutex_unlock (&iommu->lock); +return ret; +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2471,6
Quoted 82 lines
Quoted 21 lines
Quoted 1 line
+2580,55 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,Quoted 19 lines
drivers/vfio/vfio_iommu_type1.c +16
-0default :return -EINVAL ; } + + }else if (cmd ==VFIO_IOMMU_BIND ) { +struct vfio_iommu_type1_bind bind; +u32 version; +int data_size; +void *gbind_data; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_bind, flags); + +if (copy_from_user (&bind, (void __user *)arg, minsz)) +return -EFAULT ; + +if (bind.argsz < minsz) +return -EINVAL ; +Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2603
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+/* Get the version of struct iommu_gpasid_bind_data */ +if (copy_from_user (&version, + (void __user *) (arg + minsz), +sizeof (version))) +return -EFAULT ;Why are we coping things from beyond the size we've validated that the user has provided again?
let me wait for the result in Jacob's thread below. looks like need to have a decision from you and Joreg. If using argsze is good, then I guess we don't need the version-to-size mapping. right? Actually, the version-to-size mapping is added to ensure vfio copy data correctly.
https://lkml.org/lkml/2020/4/2/876Quoted 46 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2615
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ + data_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_BIND_GPASID , version); + gbind_data =kzalloc (data_size,GFP_KERNEL ); +if (!gbind_data) +return -ENOMEM ; + +if (copy_from_user (gbind_data, + (void __user *) (arg + minsz), data_size)) { +kfree (gbind_data); +return -EFAULT ; + }And again. argsz isn't just for minsz.
Quoted 31 lines
drivers/vfio/vfio_iommu_type1.c +16
-0+ +switch (bind.flags &VFIO_IOMMU_BIND_MASK ) { +case VFIO_IOMMU_BIND_GUEST_PGTBL : + ret =vfio_iommu_type1_bind_gpasid (iommu, + gbind_data); +break ; +case VFIO_IOMMU_UNBIND_GUEST_PGTBL : + ret =vfio_iommu_type1_unbind_gpasid (iommu, + gbind_data); +break ; +default : + ret = -EINVAL ; +break ; + } +kfree (gbind_data); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +1
-0@@ -14,6 +14,7 @@#include <linux/types.h> #include <linux/ioctl.h> +#include <linux/iommu.h> #define VFIO_API_VERSION 0Quoted 7 lines
Quoted 7 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 859
Quoted 7 lines
include/uapi/linux/vfio.h +3
-0@@ -853,6 +854,51 @@ struct vfio_iommu_type1_pasid_request {*/ #define VFIO_IOMMU_PASID_REQUEST _IO(VFIO_TYPE, VFIO_BASE + 22) +/** +* Supported flags: +* - VFIO_IOMMU_BIND_GUEST_PGTBL: bind guest page tables to host for
Quoted 3 lines
Quoted 3 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request {
Quoted 3 lines
include/uapi/linux/vfio.h +3
-0+* nesting type IOMMUs. In @data field It takes struct +* iommu_gpasid_bind_data. +* - VFIO_IOMMU_UNBIND_GUEST_PGTBL: undo a bind guest page table
Quoted 11 lines
Quoted 5 lines
operation
include/uapi/linux/vfio.h struct vfio_iommu_type1_pasid_request { · line 863
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* invoked by VFIO_IOMMU_BIND_GUEST_PGTBL. This must require
iommu_gpasid_bind_datain the data field as well, right?yes.
Regards,
Yi Liu
RE: [PATCH v1 6/8] vfio/type1: Bind guest page tables to host
In reply to Alex WilliamsonHi Alex,
Quoted 11 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 3:57 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 6/8] vfio/type1: Bind guest page tables to hostQuoted 3 lines · Yi L
From: Liu Yi L <[email protected]>
VFIO_TYPE1_NESTING_IOMMUis an IOMMU type which is backed by
[...]
Quoted 33 lines
Quoted 20 lines
drivers/vfio/vfio_iommu_type1.c +20
-0+/** +* Unbind specific gpasid, caller of this function requires hold +* vfio_iommu->lock +*/ +static long vfio_iommu_type1_do_guest_unbind (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +return vfio_iommu_for_each_dev (iommu, + vfio_unbind_gpasid_fn, gbind_data); +} + +static long vfio_iommu_type1_bind_gpasid (struct vfio_iommu *iommu, +struct iommu_gpasid_bind_data *gbind_data) +{ +int ret =0 ; + +mutex_lock (&iommu->lock); +if (!IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu)) { + ret = -EINVAL ; + goto out_unlock;drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2392
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ } + + ret =vfio_iommu_for_each_dev (iommu, + vfio_bind_gpasid_fn, gbind_data); +/* +* If bind failed, it may not be a total failure. Some devices +* within the iommu group may have bind successfully. Although +* we don't enable pasid capability for non-singletion iommu +* groups, a unbind operation would be helpful to ensure no +* partial binding for an iommu group. Where was the non-singleton group restriction done, I missed that.
Quoted 15 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_iommu_info_add_nesting_cap(struct vfio_iommu *iommu, · line 2401
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +9
-0+*/ +if (ret) +/* +* Undo all binds that already succeeded, no need to +* check the return value here since some device within +* the group has no successful bind when coming to this +* place switch. +*/ +vfio_iommu_type1_do_guest_unbind (iommu, gbind_data);However, the
for_each_devfunction stops when the callback function returns error, are we just assuming we stop at the same device as we faulted on the first time and that we traverse the same set of devices the second time? It seems strange to me that unbind should be able to fail.
I think the code needs enhancement. Although one group per container and one device per group is the most typical and desired case, but the code here loops domains, groups, devices. It should be able to unwind prior bind when the loop failed for a device. So I plan to do below change for bind path.
list_for_each_entry(domain, &iommu->domain_list, next) {
list_for_each_entry(group, &domain->group_list, next) {
/*
* if bind failed on a certain device, should unbind prior successful
* bind iommu_group_for_each_dev() should be modified to take two
* callbacks, one for forward loop and one for reverse loop when failure
* happened. "return_upon_failure" indicates whether return upon failure
* during forward loop or not. If yes, iommu_group_for_each_dev() should
* unwind the prior bind in this iommu group before return.
*/
ret = iommu_group_for_each_dev(iommu_group, bind_gpasid_fn,
unbind_gpasid_fn, data, return_upon_failure);
if (ret)
break;
}
if (ret) {
/* unwind bindings with prior groups */
list_for_each_entry_continue_reverse(group,
&domain->group_list, next) {
iommu_group_for_each_dev(iommu_group, unbind_gpasid_fn,
NULL, data, ignore_intermediate_failure);
}
break;
}
}if (ret) {
/* unwind bindings with prior domains */
list_for_each_entry_continue_reverse(domain, &iommu->domain_list, next) {
iommu_group_for_each_dev(iommu_group, unbind_gpasid_fn,
NULL, data, ignore_intermediate_failure);
}
}
}return ret;
Regards,
Yi Liu
PATCH 7/8 vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type VFIO_TYPE1_NESTING_IOMMU, guest "owns" the first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.
This patch adds a new IOCTL VFIO_IOMMU_CACHE_INVALIDATE to propagate guest first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU cache correctness.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
--- drivers/vfio/vfio_iommu_type1.c | 49 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)
drivers/vfio/vfio_iommu_type1.c +49 -0
@@ -2423,6 +2423,15 @@ static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu,return ret; } +static int vfio_cache_inv_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {
@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_cache_invalidate, + flags); + +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; + +if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_CACHE_INVAL , version); + + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock); + ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info); +mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;
include/uapi/linux/vfio.h +22 -0
@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, +* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE + 24) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 40 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type
VFIO_TYPE1_NESTING_IOMMU, guest "owns" the
first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.This patch adds a new IOCTL
VFIO_IOMMU_CACHE_INVALIDATEto propagate guest
first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU cache
correctness.With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 49 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)drivers/vfio/vfio_iommu_type1.c +0
-0
drivers/vfio/vfio_iommu_type1.c static long · line 2427
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +1
-0@@ -2423,6 +2423,15 @@ static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu, return ret; } +static int vfio_cache_inv_fn (struct device *dev,void *data)
vfio_iommu_cache_inv_fn
Quoted 94 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg){
drivers/vfio/vfio_iommu_type1.c +40
-0@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_cache_invalidate, + flags); + +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; + +if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_CACHE_INVAL , version); + + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock); + ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info); +mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret;drivers/vfio/vfio_iommu_type1.c +0
-0}return -ENOTTY ;include/uapi/linux/vfio.h +22
-0@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, +* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE + 24) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */
Quoted 3 lines
/*
--
2.7.4
This patch looks good to me in general. But since there is still a major open about version compatibility, I'll hold my r-b until that open is closed. 😊
Thanks
Kevin
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 47 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 8:58 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 7/8] vfio/type1: AddVFIO_IOMMU_CACHE_INVALIDATEQuoted 42 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type
VFIO_TYPE1_NESTING_IOMMU, guest "owns" the
first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.This patch adds a new IOCTL
VFIO_IOMMU_CACHE_INVALIDATEto propagate guest
first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU cache correctness.With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 49 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex a877747..937ec3f 100644drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2423,6 +2423,15 @@ static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu, return ret; }
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu, · line 2426
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+static int vfio_cache_inv_fn (struct device *dev,void *data)
vfio_iommu_cache_inv_fn
got it.
Quoted 94 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu,
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); } +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsigned long arg) { @@ -2629,6 +2638,46 @@
Quoted 84 lines
static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c +8
-0}kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct
vfio_iommu_type1_cache_invalidate,drivers/vfio/vfio_iommu_type1.c +15
-0+ flags); + +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; + +if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size = iommu_uapi_get_data_size( + IOMMU_UAPI_CACHE_INVAL,version);
drivers/vfio/vfio_iommu_type1.c +17
-0+ + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock); + ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info); +mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +23
-1@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, +* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before +getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU. +* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE + 24) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */
Quoted 7 lines
Quoted 3 lines
/*
--
2.7.4This patch looks good to me in general. But since there is still a major open about version compatibility, I'll hold my r-b until that open is closed. 😊
thanks,
Regards,
Yi Liu
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Liu, Yi Ldrivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2641
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +1
-0@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) {
Please refactor the spaghetti in this ioctl handler to use a switch statement and a helper function per command before growing it even more.
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ;
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2670
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ + info_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_CACHE_INVAL , version); + + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) {
The user might have changed the version while you were allocating and freeing the memory, introducing potentially exploitable racing conditions.
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Hi Hellwig,
Thanks for your review, Hellwig. :-) inline reply.
Quoted 7 lines
From: Christoph Hellwig <[email protected]>
Sent: Tuesday, March 31, 2020 3:56 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 7/8] vfio/type1: AddVFIO_IOMMU_CACHE_INVALIDATEReview static long vfio_iommu_type1_ioctl(void
Quoted 1 line
(quoted diff) +0
-0@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2641
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +1
-0}kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) {Please refactor the spaghetti in this ioctl handler to use a switch statement and a helper function per command before growing it even more.
got it. I may get a separate refactor patch before adding my changes.
Quoted 11 lines
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_CACHE_INVAL , version); + + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info)
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2670
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) {The user might have changed the version while you were allocating and freeing the
memory, introducing potentially exploitable racing conditions.
yeah, I know the @version is not welcomed in the thread Jacob is driving. I'll adjust the code here once the open in that thread has been solved.
But regardless of the version, I'm not sure if I 100% got your point. Could you elaborate a bit? BTW. The code somehow referenced the code below. The basic flow is copying partial data from __arg and then copy the rest data after figuring out how much left. The difference betwen below code and my code is just different way to figure out left data size. Since I'm not sure if I got your point. If the racing is true in such flow, I guess there are quite a few places need to enhance.
vfio_pci_ioctl(){
{
...
} else if (cmd == VFIO_DEVICE_SET_IRQS) {
struct vfio_irq_set hdr;
u8 *data = NULL;
int max, ret = 0;
size_t data_size = 0;
minsz = offsetofend(struct vfio_irq_set, count);
if (copy_from_user(&hdr, (void __user *)arg, minsz))
return -EFAULT;
max = vfio_pci_get_irq_count(vdev, hdr.index);
ret = vfio_set_irqs_validate_and_prepare(&hdr, max,
VFIO_PCI_NUM_IRQS, &data_size);
if (ret)
return ret;
if (data_size) {
data = memdup_user((void __user *)(arg + minsz),
data_size);
if (IS_ERR(data))
return PTR_ERR(data);
}
mutex_lock(&vdev->igate);
ret = vfio_pci_set_irqs_ioctl(vdev, hdr.flags, hdr.index,
hdr.start, hdr.count, data);
mutex_unlock(&vdev->igate);
kfree(data);
return ret;
} else if (cmd == VFIO_DEVICE_RESET) {...
}
Regards,
Yi Liu
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Liu, Yi LQuoted 50 lines · Yi L
From: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type
VFIO_TYPE1_NESTING_IOMMU, guest "owns" the first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.This patch adds a new IOCTL
VFIO_IOMMU_CACHE_INVALIDATEto propagate guest first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU cache correctness.With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 49 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)drivers/vfio/vfio_iommu_type1.c +9
-0@@ -2423,6 +2423,15 @@ static long vfio_iommu_type1_unbind_gpasid(struct vfio_iommu *iommu,return ret; } +static int vfio_cache_inv_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }kfree (gbind_data);
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2649
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +9
-0return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct vfio_iommu_type1_cache_invalidate, + flags);
This breaks backward compatibility as soon as struct iommu_cache_invalidate_info changes size by its defined versioning scheme. ie. a field gets added, the version is bumped, all existing userspace breaks. Our minsz is offsetofend to the version field, interpret the version to size, then reevaluate argsz.
Quoted 18 lines
drivers/vfio/vfio_iommu_type1.c +18
-0+ +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; + +if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size =iommu_uapi_get_data_size ( +IOMMU_UAPI_CACHE_INVAL , version); + + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ;
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2677
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock); + ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info);
How does a user respond when their cache invalidate fails? Isn't this also another case where our for_each_dev can fail at an arbitrary point leaving us with no idea whether each device even had the opportunity to perform the invalidation request. I don't see how we have any chance to maintain coherency after this faults.
Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +2
-0@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24,
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind { · line 913
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU.
Is this a necessary qualifier? A user can try to call this ioctl at any point, it only makes sense in certain configurations, but it should always "do the right thing" relative to the container iommu config.
Also, I don't see anything in these last few patches testing the operating IOMMU model, what happens when a user calls them when not using the nesting IOMMU?
Is this ioctl and the previous BIND ioctl only valid when configured for the nesting IOMMU type?
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind { · line 922
Quoted 9 lines
include/uapi/linux/vfio.h +9
-0+* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE + 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as CACHE_INFO.
Quoted 4 lines
include/uapi/linux/vfio.h +1
-0+/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 73 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 4:24 AMQuoted 3 lines · Yi L
From: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type
VFIO_TYPE1_NESTING_IOMMU, guest"owns" the
Quoted 5 lines
first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.
This patch adds a new IOCTL
VFIO_IOMMU_CACHE_INVALIDATEtopropagate guest
Quoted 1 line
first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU
cache
Quoted 13 lines
correctness.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio_iommu_type1.c | 49+++++++++++++++++++++++++++++++++++++++++
Review
Quoted 4 lines
include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cdrivers/vfio/vfio_iommu_type1.c static long
Quoted 4 lines
index a877747..937ec3f 100644
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2423,6 +2423,15 @@ static longvfio_iommu_type1_unbind_gpasid(struct
vfio_iommu*iommu,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +9
-0return ret; } +static int vfio_cache_inv_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +8
-0}kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct
vfio_iommu_type1_cache_invalidate,
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2649
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ flags);This breaks backward compatibility as soon as struct
iommu_cache_invalidate_infochanges size by its defined versioning scheme. ie. a field gets added, the version is bumped, all existing userspace breaks. Our minsz is offsetofend to the version field, interpret the version to size, then reevaluate argsz.
btw the version scheme is challenged by Christoph Hellwig. After some discussions, we need your guidance how to move forward. Jacob summarized available options below:
https://lkml.org/lkml/2020/4/2/876
Quoted 28 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; +drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size = iommu_uapi_get_data_size( + IOMMU_UAPI_CACHE_INVAL,version);
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock);
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2677
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info);How does a user respond when their cache invalidate fails? Isn't this also another case where our
for_each_devcan fail at an arbitrary point leaving us with no idea whether each device even had the opportunity to perform the invalidation request. I don't see how we have any chance to maintain coherency after this faults.
Then can we make it simple to support singleton group only?
Quoted 32 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +2
-0@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, include/uapi/linux/vfio.h struct vfio_iommu_type1_bind { · line 913
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU. Is this a necessary qualifier? A user can try to call this ioctl at any point, it only makes sense in certain configurations, but it should always "do the right thing" relative to the container iommu config.
Quoted 6 lines
Also, I don't see anything in these last few patches testing the operating IOMMU model, what happens when a user calls them when not using the nesting IOMMU?
Is this ioctl and the previous BIND ioctl only valid when configured for the nesting IOMMU type?
I think so. We should add the nesting check in those new ioctls.
Quoted 12 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 10 lines
include/uapi/linux/vfio.h +9
-0+* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE+ 24)
Quoted 2 lines
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.
Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide version_to_size conversion and instead we just pass data[] to iommu driver for further audit?
Quoted 6 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind { · line 924
Quoted 3 lines
include/uapi/linux/vfio.h +1
-0+/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */
Quoted 2 lines
include/uapi/linux/vfio.h +0
-0/*
Thanks
Kevin
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 85 lines · Kevin
Quoted 77 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 4:24 AMQuoted 3 lines · Yi L
From: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type
VFIO_TYPE1_NESTING_IOMMU, guest"owns" the
Quoted 6 lines
first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.
This patch adds a new IOCTL
VFIO_IOMMU_CACHE_INVALIDATEtopropagate guest
Quoted 2 lines
first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU
cache
Quoted 13 lines
correctness.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio_iommu_type1.c | 49+++++++++++++++++++++++++++++++++++++++++
Review
Quoted 4 lines
include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cdrivers/vfio/vfio_iommu_type1.c static long
Quoted 4 lines
index a877747..937ec3f 100644
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2423,6 +2423,15 @@ static longvfio_iommu_type1_unbind_gpasid(struct
vfio_iommu*iommu,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +8
-0return ret; } +static int vfio_cache_inv_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +} +static long vfio_iommu_type1_ioctl (void *iommu_data, unsigned int cmd, unsignedlong arg) {
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +8
-0}kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct
vfio_iommu_type1_cache_invalidate,Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2649
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ flags);This breaks backward compatibility as soon as struct
iommu_cache_invalidate_infochanges size by its defined versioning scheme. ie. a field gets added, the version is bumped, all existing userspace breaks. Our minsz is offsetofend to the version field, interpret the version to size, then reevaluate argsz.
Quoted 5 lines
btw the version scheme is challenged by Christoph Hellwig. After some discussions, we need your guidance how to move forward. Jacob summarized available options below:
https://lkml.org/lkml/2020/4/2/876
For this particular case, I don't quite get the difference between minsz=flags and minsz=version. Our IOMMU version scheme will only change size at the end where the variable union is used for vendor specific extensions. Version bump does not change size (only re-purpose padding) from the start of the UAPI structure to the union, i.e. version will __always__ be after struct vfio_iommu_type1_cache_invalidate.flags.
Quoted 84 lines
Quoted 34 lines
Quoted 14 lines
drivers/vfio/vfio_iommu_type1.c +8
-0+ +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; + +if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct
iommu_cache_invalidate_info*/drivers/vfio/vfio_iommu_type1.c +2
-0+ if (copy_from_user(&version, + (void __user *) (arg + minsz),sizeof(version)))
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+return -EFAULT ; + + info_size = iommu_uapi_get_data_size( + IOMMU_UAPI_CACHE_INVAL,version);
Quoted 15 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + + if (copy_from_user(cache_info, + (void __user *) (arg + minsz),
info_size)) {drivers/vfio/vfio_iommu_type1.c +6
-0+kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock); + ret = vfio_iommu_for_each_dev(iommu,
vfio_cache_inv_fn,Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2677
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ cache_info);How does a user respond when their cache invalidate fails? Isn't this also another case where our
for_each_devcan fail at an arbitrary point leaving us with no idea whether each device even had the opportunity to perform the invalidation request. I don't see how we have any chance to maintain coherency after this faults.Then can we make it simple to support singleton group only?
Quoted 39 lines
Quoted 25 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +6
-0@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, +* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache
+ * invalidation information is conveyed by @cache_info, the contentQuoted 8 lines
+ * format would be structures defined in
uapi/linux/iommu.h. User
+ * should be aware of that the structiommu_cache_invalidate_info
+ * has a @version field, vfio needs to parse this field before getting
+ * data from userspace.
+ *
+ * Availability of this IOCTL is afterVFIO_SET_IOMMU.Is this a necessary qualifier? A user can try to call this ioctl at any point, it only makes sense in certain configurations, but it should always "do the right thing" relative to the container iommu config.
Quoted 9 lines
Quoted 6 lines
Also, I don't see anything in these last few patches testing the operating IOMMU model, what happens when a user calls them when not using the nesting IOMMU?
Is this ioctl and the previous BIND ioctl only valid when configured for the nesting IOMMU type?
I think so. We should add the nesting check in those new ioctls.
I also added nesting domain attribute check in IOMMU driver, so bind guest PASID will fail if nesting mode is not supported. There will be no invalidation w/o bind.
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASE+ 24)
Quoted 8 lines
Quoted 3 lines
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?
I guess Alex meant we do something similar to:
struct vfio_irq_set {
__u32 argsz;
__u32 flags;#define VFIO_IRQ_SET_DATA_NONE (1 << 0) /* Data not present */
#define VFIO_IRQ_SET_DATA_BOOL (1 << 1) /* Data is bool (u8) */
#define VFIO_IRQ_SET_DATA_EVENTFD (1 << 2) /* Data is eventfd (s32) */
#define VFIO_IRQ_SET_ACTION_MASK (1 << 3) /* Mask interrupt */
#define VFIO_IRQ_SET_ACTION_UNMASK (1 << 4) /* Unmask interrupt */
#define VFIO_IRQ_SET_ACTION_TRIGGER (1 << 5) /* Trigger interrupt */ __u32 index; __u32 start; __u32 count; __u8 data[]; };
So we could do:
struct vfio_iommu_type1_cache_invalidate {
__u32 argsz;
#define VFIO_INVL_DATA_NONE
#define VFIO_INVL_DATA_CACHE_INFO (1 << 1)
__u32 flags;
__u8 data[];
}We still need version_to_size version, but under
if (flag & VFIO_INVL_DATA_CACHE_INFO)
get_size_from_version();
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
+
/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU
--------*/
Quoted 2 lines
/*
Thanks
Kevin
[Jacob Pan]
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Tian, KevinQuoted 81 lines · Kevin
Quoted 73 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 4:24 AMQuoted 3 lines · Yi L
From: Liu Yi L <[email protected]>
For VFIO IOMMUs with the type
VFIO_TYPE1_NESTING_IOMMU, guest"owns" the
Quoted 5 lines
first-level/stage-1 translation structures, the host IOMMU driver has no knowledge of first-level/stage-1 structure cache updates unless the guest invalidation requests are trapped and propagated to the host.
This patch adds a new IOCTL
VFIO_IOMMU_CACHE_INVALIDATEtopropagate guest
Quoted 1 line
first-level/stage-1 IOMMU cache invalidations to host to ensure IOMMU
cache
Quoted 13 lines
correctness.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used safely as the host IOMMU iotlb correctness are ensured.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>Signed-off-by Eric Auger <[email protected]>Signed-off-by Jacob Pan <[email protected]>Quoted 2 lines
--- drivers/vfio/vfio_iommu_type1.c | 49+++++++++++++++++++++++++++++++++++++++++
Review
Quoted 4 lines
include/uapi/linux/vfio.h | 22 ++++++++++++++++++ 2 files changed, 71 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cdrivers/vfio/vfio_iommu_type1.c static long
Quoted 4 lines
index a877747..937ec3f 100644
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2423,6 +2423,15 @@ static longvfio_iommu_type1_unbind_gpasid(struct
vfio_iommu*iommu,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void
Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +9
-0return ret; } +static int vfio_cache_inv_fn (struct device *dev,void *data) +{ +struct domain_capsule *dc = (struct domain_capsule *)data; +struct iommu_cache_invalidate_info *cache_inv_info = + (struct iommu_cache_invalidate_info *) dc->data; + +return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {@@ -2629,6 +2638,46 @@ static long vfio_iommu_type1_ioctl(void
*iommu_data,drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 11 lines
drivers/vfio/vfio_iommu_type1.c +8
-0}kfree (gbind_data);return ret; + }else if (cmd ==VFIO_IOMMU_CACHE_INVALIDATE ) { +struct vfio_iommu_type1_cache_invalidate cache_inv; +u32 version; +int info_size; +void *cache_info; +int ret; + + minsz = offsetofend(struct
vfio_iommu_type1_cache_invalidate,Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2649
Quoted 1 line
drivers/vfio/vfio_iommu_type1.c +1
-0+ flags);This breaks backward compatibility as soon as struct
iommu_cache_invalidate_infochanges size by its defined versioning scheme. ie. a field gets added, the version is bumped, all existing userspace breaks. Our minsz is offsetofend to the version field, interpret the version to size, then reevaluate argsz.
Quoted 4 lines
btw the version scheme is challenged by Christoph Hellwig. After some discussions, we need your guidance how to move forward. Jacob summarized available options below:
https://lkml.org/lkml/2020/4/2/876
Ok
Quoted 37 lines
Quoted 28 lines
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; +drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size = iommu_uapi_get_data_size( + IOMMU_UAPI_CACHE_INVAL,version);
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c +12
-0+ + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ; + +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock);Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2677
Quoted 2 lines
drivers/vfio/vfio_iommu_type1.c +2
-0+ ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info);How does a user respond when their cache invalidate fails? Isn't this also another case where our
for_each_devcan fail at an arbitrary point leaving us with no idea whether each device even had the opportunity to perform the invalidation request. I don't see how we have any chance to maintain coherency after this faults.
Quoted 1 line
Then can we make it simple to support singleton group only?
Are you suggesting a single group per container or a single device per group? Unless we have both, aren't we always going to have this issue. OTOH, why should a cache invalidate fail?
Quoted 52 lines
Quoted 31 lines
Quoted 16 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +2
-0@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, include/uapi/linux/vfio.h struct vfio_iommu_type1_bind { · line 913
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU. Is this a necessary qualifier? A user can try to call this ioctl at any point, it only makes sense in certain configurations, but it should always "do the right thing" relative to the container iommu config.
Quoted 6 lines
Also, I don't see anything in these last few patches testing the operating IOMMU model, what happens when a user calls them when not using the nesting IOMMU?
Is this ioctl and the previous BIND ioctl only valid when configured for the nesting IOMMU type?
I think so. We should add the nesting check in those new ioctls.
Quoted 12 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 10 lines
include/uapi/linux/vfio.h +9
-0+* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE+ 24)
Quoted 6 lines
Quoted 2 lines
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace cache_info with data[] then we can define a flag to specify that data[] is struct iommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expanding cache_info to meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,
Alex
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Hi Alex,
Quoted 15 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM
Subject: Re: [PATCH v1 7/8] vfio/type1: AddVFIO_IOMMU_CACHE_INVALIDATEQuoted 8 lines · Kevin
Quoted 8 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 4:24 AMQuoted 2 lines · Yi L
From: Liu Yi L <[email protected]>
[...]
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ +if (copy_from_user (&cache_inv, (void __user *)arg, minsz)) +return -EFAULT ; +drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if (cache_inv.argsz < minsz || cache_inv.flags) +return -EINVAL ; + +/* Get the version of struct iommu_cache_invalidate_info */ +if (copy_from_user (&version, + (void __user *) (arg + minsz),sizeof (version))) +return -EFAULT ; + + info_size = iommu_uapi_get_data_size( + IOMMU_UAPI_CACHE_INVAL,version);
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +4
-0+ + cache_info =kzalloc (info_size,GFP_KERNEL ); +if (!cache_info) +return -ENOMEM ;drivers/vfio/vfio_iommu_type1.c static long vfio_iommu_type1_ioctl(void *iommu_data, · line 2677
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+ +if (copy_from_user (cache_info, + (void __user *) (arg + minsz), info_size)) { +kfree (cache_info); +return -EFAULT ; + } + +mutex_lock (&iommu->lock); + ret =vfio_iommu_for_each_dev (iommu, vfio_cache_inv_fn, + cache_info);How does a user respond when their cache invalidate fails? Isn't this also another case where our
for_each_devcan fail at an arbitrary point leaving us with no idea whether each device even had the opportunity to perform the invalidation request. I don't see how we have any chance to maintain coherency after this faults.
Quoted 4 lines
Quoted 1 line
Then can we make it simple to support singleton group only?
Are you suggesting a single group per container or a single device per group? Unless we have both, aren't we always going to have this issue.
Agreed. we need both to avoid the potential for_each_dev() loop issue. I suppose this is also the most typical and desired config for vSVA support. I think it makes sense with below items:
a) one group per container
PASID and nested translation gives user-space a chance to attach their page table (e.g. guest process page table) to host IOMMU, this is vSVA. If adding multiple groups to a vSVA-capable container, then a SVA bind on this container means bind it with all groups (devices are included) within the container. This doesn't make sense with three reasons: for one the passthru devices are not necessary to be manipulated by same guest application; for two passthru devices are not surely added in a single guest group; for three not all passthru devices (either from different group or same group) are sva capable.
As above, enforce one group per container makes sense to me.
b) one device per group
SVA support is limited to singleton group so far in bare-metal bind per Jean's series. I think it's be good to follow it in passthru case.
https://patchwork.kernel.org/patch/10213877/
https://lkml.org/lkml/2019/4/10/663
As mentioned in a), group may have both SVA-capable device and non-SVA -capable device, it would be a problem for VFIO to figure a way to isolate them.
Quoted 1 line
OTOH, why should a cache invalidate fail?
there are sanity check done by vendor iommu driver against the invalidate request from userspace. so it may fail if sanity check failed. But I guess it may be better to something like abort instead of fail the request. isn't?
Quoted 60 lines
Quoted 53 lines
Quoted 32 lines
Quoted 17 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+mutex_unlock (&iommu->lock); +kfree (cache_info); +return ret; }return -ENOTTY ;include/uapi/linux/vfio.h +2
-0@@ -899,6 +899,28 @@ struct vfio_iommu_type1_bind {*/ #define VFIO_IOMMU_BIND _IO(VFIO_TYPE, VFIO_BASE + 23) +/** +* VFIO_IOMMU_CACHE_INVALIDATE - _IOW(VFIO_TYPE, VFIO_BASE + 24, include/uapi/linux/vfio.h struct vfio_iommu_type1_bind { · line 913
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+* struct vfio_iommu_type1_cache_invalidate) +* +* Propagate guest IOMMU cache invalidation to the host. The cache +* invalidation information is conveyed by @cache_info, the content +* format would be structures defined in uapi/linux/iommu.h. User +* should be aware of that the struct iommu_cache_invalidate_info +* has a @version field, vfio needs to parse this field before getting +* data from userspace. +* +* Availability of this IOCTL is after VFIO_SET_IOMMU. Is this a necessary qualifier? A user can try to call this ioctl at any point, it only makes sense in certain configurations, but it should always "do the right thing" relative to the container iommu config.
Quoted 6 lines
Also, I don't see anything in these last few patches testing the operating IOMMU model, what happens when a user calls them when not using the nesting IOMMU?
Is this ioctl and the previous BIND ioctl only valid when configured for the nesting IOMMU type?
I think so. We should add the nesting check in those new ioctls.
Quoted 12 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 10 lines
include/uapi/linux/vfio.h +9
-0+* +* returns: 0 on success, -errno on failure. +*/ +struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE, VFIO_BASE+ 24)
Quoted 6 lines
Quoted 2 lines
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?
Quoted 11 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though.
Ok, I think data[] is acceptable. BTW. Do you have any decision on the uapi version open iin Jacob's thread? I'd like to re-work my patch based on your decision.
https://lkml.org/lkml/2020/4/2/876
thanks again for your help. :-)
Regards,
Yi Liu
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Alex WilliamsonHi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.
Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM
[...]
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 8 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?
Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,
Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/ unbind_gpasid, cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid and cache_inv as both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?
ioctl #22: VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags == ALLOC_PASID
* specify a pasid to be freed when flags == FREE_PASID
* @range: specify the allocation range when flags == ALLOC_PASID */
struct vfio_iommu_pasid_request {
__u32 argsz;
#define VFIO_IOMMU_ALLOC_PASID (1 << 0)
#define VFIO_IOMMU_FREE_PASID (1 << 1)
__u32 flags;
__u32 pasid;
struct {
__u32 min;
__u32 max;
} range;
};ioctl #23: VFIO_IOMMU_NESTING_OP
struct vfio_iommu_type1_nesting_op {
__u32 argsz;
__u32 flags;
__u32 op;
__u8 data[];
};/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2
Thanks,
Yi Liu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 78 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, April 16, 2020 6:40 PMHi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 4 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Quoted 5 lines
Quoted 4 lines
Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==ALLOC_PASID
* specify a pasid to be freed when flags ==FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/struct vfio_iommu_pasid_request { __u32 argsz; #define VFIO_IOMMU_ALLOC_PASID (1 << 0) #define VFIO_IOMMU_FREE_PASID (1 << 1) __u32 flags; __u32 pasid; struct { __u32 min; __u32 max; } range; };ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };
Quoted 5 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2
Then why cannot we just put PASID into the header since the majority of nested usage is associated with a pasid?
ioctl #23: VFIO_IOMMU_NESTING_OP
struct vfio_iommu_type1_nesting_op {
__u32 argsz;
__u32 flags;
__u32 op;
__u32 pasid;
__u8 data[];
};In case of SMMUv2 which supports nested w/o PASID, this field can be ignored for that specific case.
Thanks
Kevin
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Hi Kevin,
Quoted 96 lines · Kevin
Quoted 78 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, April 16, 2020 6:40 PMHi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 4 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Quoted 5 lines
Quoted 4 lines
Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==ALLOC_PASID
* specify a pasid to be freed when flags ==FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/struct vfio_iommu_pasid_request { __u32 argsz; #define VFIO_IOMMU_ALLOC_PASID (1 << 0) #define VFIO_IOMMU_FREE_PASID (1 << 1) __u32 flags; __u32 pasid; struct { __u32 min; __u32 max; } range; };ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };Quoted 5 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2Then why cannot we just put PASID into the header since the majority of nested usage is associated with a pasid?
ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u32 pasid; __u8 data[]; };
Quoted 2 lines
In case of SMMUv2 which supports nested w/o PASID, this field can be ignored for that specific case.
On my side I would prefer keeping the pasid in the data[]. This is not always used.
For instance, in iommu_cache_invalidate_info/iommu_inv_pasid_info we devised flags to tell whether the PASID is used.
Thanks
Eric
Quoted 4 lines
Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 102 lines
From: Auger Eric <[email protected]>
Sent: Thursday, April 16, 2020 8:43 PMHi Kevin,
Quoted 52 lines · Kevin
Quoted 52 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, April 16, 2020 6:40 PMHi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines
Review
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */(quoted diff) +6
-0+struct vfio_iommu_type1_cache_invalidate { + __u32 argsz; + __u32 flags; + struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 4 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Quoted 5 lines
Quoted 4 lines
Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?Quoted 3 lines
Quoted 3 lines
ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==
ALLOC_PASIDQuoted 41 lines
Quoted 23 lines
* specify a pasid to be freed when flags ==
FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/struct vfio_iommu_pasid_request { __u32 argsz; #define VFIO_IOMMU_ALLOC_PASID (1 << 0) #define VFIO_IOMMU_FREE_PASID (1 << 1) __u32 flags; __u32 pasid; struct { __u32 min; __u32 max; } range; };ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };Quoted 5 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2Then why cannot we just put PASID into the header since the majority of nested usage is associated with a pasid?
ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u32 pasid; __u8 data[]; };
Quoted 8 lines
Quoted 2 lines
In case of SMMUv2 which supports nested w/o PASID, this field can be ignored for that specific case.
On my side I would prefer keeping the pasid in the data[]. This is not always used.
For instance, in iommu_cache_invalidate_info/iommu_inv_pasid_info we devised flags to tell whether the PASID is used.
But don't we include a PASID in both invalidate structures already?
struct iommu_inv_addr_info {
#define IOMMU_INV_ADDR_FLAGS_PASID (1 << 0)
#define IOMMU_INV_ADDR_FLAGS_ARCHID (1 << 1)
#define IOMMU_INV_ADDR_FLAGS_LEAF (1 << 2)
__u32 flags;
__u32 archid;
__u64 pasid;
__u64 addr;
__u64 granule_size;
__u64 nb_granules;};
struct iommu_inv_pasid_info {
#define IOMMU_INV_PASID_FLAGS_PASID (1 << 0)
#define IOMMU_INV_PASID_FLAGS_ARCHID (1 << 1)
__u32 flags;
__u32 archid;
__u64 pasid;
};then consolidating the pasid field into generic header doesn't hurt. the specific handler still rely on flags to tell whether it is used?
Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Hi Kevin,
Quoted 104 lines · Kevin
Quoted 104 lines
From: Auger Eric <[email protected]>
Sent: Thursday, April 16, 2020 8:43 PMHi Kevin,
Quoted 52 lines · Kevin
Quoted 52 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, April 16, 2020 6:40 PMHi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines
Review
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */(quoted diff) +6
-0+struct vfio_iommu_type1_cache_invalidate { + __u32 argsz; + __u32 flags; + struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 4 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Quoted 5 lines
Quoted 4 lines
Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?Quoted 3 lines
Quoted 3 lines
ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==
ALLOC_PASIDQuoted 43 lines
Quoted 23 lines
* specify a pasid to be freed when flags ==
FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/struct vfio_iommu_pasid_request { __u32 argsz; #define VFIO_IOMMU_ALLOC_PASID (1 << 0) #define VFIO_IOMMU_FREE_PASID (1 << 1) __u32 flags; __u32 pasid; struct { __u32 min; __u32 max; } range; };ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };Quoted 5 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2Then why cannot we just put PASID into the header since the majority of nested usage is associated with a pasid?
ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u32 pasid; __u8 data[]; };In case of SMMUv2 which supports nested w/o PASID, this field can be ignored for that specific case.
Quoted 8 lines
Quoted 6 lines
On my side I would prefer keeping the pasid in the data[]. This is not always used.
For instance, in iommu_cache_invalidate_info/iommu_inv_pasid_info we devised flags to tell whether the PASID is used.
But don't we include a PASID in both invalidate structures already?
The pasid presence is indicated by the IOMMU_INV_ADDR_FLAGS_PASID flag.
For instance for nested stage SMMUv3 I current performs an ARCHID (asid) based invalidation only.
Eric
Quoted 21 lines
struct
iommu_inv_addr_info{#define IOMMU_INV_ADDR_FLAGS_PASID (1 << 0) #define IOMMU_INV_ADDR_FLAGS_ARCHID (1 << 1) #define IOMMU_INV_ADDR_FLAGS_LEAF (1 << 2) __u32 flags; __u32 archid; __u64 pasid; __u64 addr; __u64 granule_size; __u64 nb_granules;};
struct
iommu_inv_pasid_info{
#defineIOMMU_INV_PASID_FLAGS_PASID(1 << 0)#define IOMMU_INV_PASID_FLAGS_ARCHID (1 << 1) __u32 flags; __u32 archid; __u64 pasid; };
Quoted 7 lines
then consolidating the pasid field into generic header doesn't hurt. the specific handler still rely on flags to tell whether it is used?
Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Liu, Yi LQuoted 54 lines · Yi L
Hi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 8 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==ALLOC_PASID
* specify a pasid to be freed when flags ==FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/
Quoted 11 lines
struct vfio_iommu_pasid_request {
__u32 argsz;
#define VFIO_IOMMU_ALLOC_PASID (1 << 0)
#define VFIO_IOMMU_FREE_PASID (1 << 1)
__u32 flags;
__u32 pasid;
struct {
__u32 min;
__u32 max;
} range;
};Can't the ioctl return the pasid valid on alloc (like GET_DEVICE_FD)? Would it be useful to support freeing a range of pasids? If so then we could simply use range for both, ie. allocate a pasid from this range and return it, or free all pasids in this range? vfio already needs to track pasids to free them on release, so presumably this is something we could support easily.
Quoted 7 lines
ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };
data only has 4-byte alignment, I think we really want it at an 8-byte alignment. This is why I embedded the "op" into the flag for DEVICE_FEATURE. Thanks,
Alex
Quoted 9 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2Thanks,
Yi Liu
Re: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
Quoted 69 lines · On Thu, 16 Apr 2020 08:40:31 -0600 Alex Williamson…
Quoted 54 lines · Yi L
Hi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 8 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==ALLOC_PASID
* specify a pasid to be freed when flags ==FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/Quoted 11 lines
struct vfio_iommu_pasid_request { __u32 argsz; #define VFIO_IOMMU_ALLOC_PASID (1 << 0) #define VFIO_IOMMU_FREE_PASID (1 << 1) __u32 flags; __u32 pasid; struct { __u32 min; __u32 max; } range; };
Quoted 1 line
Can't the ioctl return the pasid valid on alloc (like
GET_DEVICE_FD)?
s/valid/value/
Quoted 26 lines
Would it be useful to support freeing a range of pasids? If so then we could simply use range for both, ie. allocate a pasid from this range and return it, or free all pasids in this range? vfio already needs to track pasids to free them on release, so presumably this is something we could support easily.
Quoted 7 lines
ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };data only has 4-byte alignment, I think we really want it at an 8-byte alignment. This is why I embedded the "op" into the flag for
DEVICE_FEATURE. Thanks,Alex
Quoted 6 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2
Quoted 9 lines
Quoted 3 lines
Thanks,
Yi Liu_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 7/8] vfio/type1: Add VFIO_IOMMU_CACHE_INVALIDATE
In reply to Alex WilliamsonHi Alex,
Quoted 74 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, April 16, 2020 10:41 PM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 7/8] vfio/type1: AddVFIO_IOMMU_CACHE_INVALIDATEQuoted 54 lines · Yi L
Hi Alex,
Still have a direction question with you. Better get agreement with you before heading forward.Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 11:35 PM[...]
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_bind {
Quoted 9 lines
+ *
+ * returns: 0 on success, -errno on failure.
+ */include/uapi/linux/vfio.h +6
-0+struct vfio_iommu_type1_cache_invalidate { +__u32 argsz; +__u32 flags; +struct iommu_cache_invalidate_info cache_info; +}; +#define VFIO_IOMMU_CACHE_INVALIDATE _IO(VFIO_TYPE,
VFIO_BASEQuoted 8 lines
Quoted 4 lines
+ 24)
The future extension capabilities of this ioctl worry me, I wonder if we should do another data[] with flag defining that data as
CACHE_INFO.Can you elaborate? Does it mean with this way we don't rely on iommu driver to provide
version_to_sizeconversion and instead we just pass data[] to iommu driver for further audit?Quoted 12 lines
No, my concern is that this ioctl has a single function, strictly tied to the iommu uapi. If we replace
cache_infowith data[] then we can define a flag to specify that data[] is structiommu_cache_invalidate_info, and if we need to, a different flag to identify data[] as something else. For example if we get stuck expandingcache_infoto meet new demands and develop a new uapi to solve that, how would we expand this ioctl to support it rather than also create a new ioctl? There's also a trade-off in making the ioctl usage more difficult for the user. I'd still expect the vfio layer to check the flag and interpret data[] as indicated by the flag rather than just passing a blob of opaque data to the iommu layer though. Thanks,Based on your comments about defining a single ioctl and a unified vfio structure (with a @data[] field) for pasid_alloc/free, bind/
unbind_gpasid,cache_inv. After some offline trying, I think it would be good for bind/unbind_gpasid andcache_invas both of them use the iommu uapi definition. While the pasid alloc/free operation doesn't. It would be weird to put all of them together. So pasid alloc/free may have a separate ioctl. It would look as below. Does this direction look good per your opinion?ioctl #22:
VFIO_IOMMU_PASID_REQUEST
/**
* @pasid: used to return the pasid alloc result when flags ==ALLOC_PASID
* specify a pasid to be freed when flags ==FREE_PASID
* @range: specify the allocation range when flags ==ALLOC_PASID*/Quoted 11 lines
struct vfio_iommu_pasid_request { __u32 argsz; #define VFIO_IOMMU_ALLOC_PASID (1 << 0) #define VFIO_IOMMU_FREE_PASID (1 << 1) __u32 flags; __u32 pasid; struct { __u32 min; __u32 max; } range; };
Quoted 1 line
Can't the ioctl return the pasid valid on alloc (like
GET_DEVICE_FD)?
Yep, I think you mentioned before. At that time, I believed it would be better to return the result via a __u32 buffer so that make full use of the 32 bits. But looks like it doesn't make much difference. I'll follow your suggestion.
Quoted 5 lines
Would it be useful to support freeing a range of pasids? If so then we could simply use range for both, ie. allocate a pasid from this range and return it, or free all pasids in this range? vfio already needs to track pasids to free them on release, so presumably this is something we could support easily.
yes, I think it is a nice thing. then I can remove the @pasid field. will do it.
Quoted 11 lines
Quoted 7 lines
ioctl #23:
VFIO_IOMMU_NESTING_OPstruct vfio_iommu_type1_nesting_op { __u32 argsz; __u32 flags; __u32 op; __u8 data[]; };data only has 4-byte alignment, I think we really want it at an 8-byte alignment. This is why I embedded the "op" into the flag for
DEVICE_FEATURE. Thanks,
got it. I may also merge the op into flags (maybe the lower 16 bits for op).
Thanks,
Yi Liu
Quoted 11 lines
Alex
Quoted 9 lines
/* Nesting Ops */
#define VFIO_IOMMU_NESTING_OP_BIND_PGTBL 0 #define VFIO_IOMMU_NESTING_OP_UNBIND_PGTBL 1 #define VFIO_IOMMU_NESTING_OP_CACHE_INVLD 2Thanks,
Yi Liu
PATCH 8/8 vfio/type1: Add vSVA support for IOMMU-backed mdevs
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
Recent years, mediated device pass-through framework (e.g. vfio-mdev) are used to achieve flexible device sharing across domains (e.g. VMs). Also there are hardware assisted mediated pass-through solutions from platform vendors. e.g. Intel VT-d scalable mode which supports Intel Scalable I/O Virtualization technology. Such mdevs are called IOMMU- backed mdevs as there are IOMMU enforced DMA isolation for such mdevs. In kernel, IOMMU-backed mdevs are exposed to IOMMU layer by aux-domain concept, which means mdevs are protected by an iommu domain which is aux-domain of its physical device. Details can be found in the KVM presentation from Kevin Tian. IOMMU-backed equals to IOMMU-capable.
https://events19.linuxfoundation.org/wp-content/uploads/2017/12/\
Hardware-Assisted-Mediated-Pass-Through-with-VFIO-Kevin-Tian-Intel.pdf
This patch supports NESTING IOMMU for IOMMU-backed mdevs by figuring out the physical device of an IOMMU-backed mdev and then invoking IOMMU requests to IOMMU layer with the physical device and the mdev's aux domain info.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used on IOMMU-backed mdevs.
--- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
drivers/vfio/vfio_iommu_type1.c +20 -3
@@ -132,6 +132,7 @@ struct vfio_regions {struct domain_capsule {struct iommu_domain *domain; +struct vfio_group *group;void *data; };
@@ -148,6 +149,7 @@ static int vfio_iommu_for_each_dev(struct vfio_iommu *iommu,list_for_each_entry (d, &iommu->domain_list, next) { dc.domain = d->domain;list_for_each_entry (g, &d->group_list, next) { + dc.group = g; ret =iommu_group_for_each_dev (g->iommu_group, &dc, fn);if (ret)
@@ -2347,7 +2349,12 @@ static int vfio_bind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +if (dc->group->mdev_group) +return iommu_sva_bind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data); +else +return iommu_sva_bind_gpasid (dc->domain, + dev, gbind_data); }static int vfio_unbind_gpasid_fn (struct device *dev,void *data)
@@ -2356,8 +2363,13 @@ static int vfio_unbind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_unbind_gpasid (dc->domain, dev, +if (dc->group->mdev_group) +return iommu_sva_unbind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data->hpasid); +else +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); }/**
@@ -2429,7 +2441,12 @@ static int vfio_cache_inv_fn(struct device *dev, void *data)struct iommu_cache_invalidate_info *cache_inv_info = (struct iommu_cache_invalidate_info *) dc->data; -return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +if (dc->group->mdev_group) +return iommu_cache_invalidate (dc->domain, +vfio_mdev_get_iommu_device (dev), cache_inv_info); +else +return iommu_cache_invalidate (dc->domain, + dev, cache_inv_info); } static long vfio_iommu_type1_ioctl(void *iommu_data,
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 8/8] vfio/type1: Add vSVA support for IOMMU-backed mdevs
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PMFrom: Liu Yi L <[email protected]>
Recent years, mediated device pass-through framework (e.g. vfio-mdev) are used to achieve flexible device sharing across domains (e.g. VMs).
are->is
Quoted 8 lines
Also there are hardware assisted mediated pass-through solutions from platform vendors. e.g. Intel VT-d scalable mode which supports Intel Scalable I/O Virtualization technology. Such mdevs are called IOMMU- backed mdevs as there are IOMMU enforced DMA isolation for such mdevs. In kernel, IOMMU-backed mdevs are exposed to IOMMU layer by aux- domain
concept, which means mdevs are protected by an iommu domain which is aux-domain of its physical device. Details can be found in the KVM
"by an iommu domain which is auxiliary to the domain that the kernel driver primarily uses for DMA API"
Quoted 10 lines
presentation from Kevin Tian. IOMMU-backed equals to IOMMU-capable.
https://events19.linuxfoundation.org/wp-content/uploads/2017/12/\
Hardware-Assisted-Mediated-Pass-Through-with-VFIO-Kevin-Tian-Intel.pdfThis patch supports NESTING IOMMU for IOMMU-backed mdevs by figuring out the physical device of an IOMMU-backed mdev and then invoking IOMMU
requests to IOMMU layer with the physical device and the mdev's aux domain info.
"and then calling into the IOMMU layer to complete the vSVA operations on the aux domain associated with that mdev"
Quoted 71 lines
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used on IOMMU-backed mdevs.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>CC Jun Tian <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)drivers/vfio/vfio_iommu_type1.c +2
-0@@ -132,6 +132,7 @@ struct vfio_regions {struct domain_capsule {struct iommu_domain *domain; +struct vfio_group *group;void *data; };@@ -148,6 +149,7 @@ static int vfio_iommu_for_each_dev(struct vfio_iommu *iommu, list_for_each_entry(d, &iommu->domain_list, next) { dc.domain = d->domain;list_for_each_entry (g, &d->group_list, next) { + dc.group = g; ret =iommu_group_for_each_dev (g->iommu_group, &dc, fn);drivers/vfio/vfio_iommu_type1.c +6
-1if (ret)@@ -2347,7 +2349,12 @@ static int vfio_bind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +if (dc->group->mdev_group) +return iommu_sva_bind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data); +else +return iommu_sva_bind_gpasid (dc->domain, + dev, gbind_data); }static int
vfio_unbind_gpasid_fn(struct device *dev, void *data)drivers/vfio/vfio_iommu_type1.c +6
-1@@ -2356,8 +2363,13 @@ static int vfio_unbind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_unbind_gpasid (dc->domain, dev, +if (dc->group->mdev_group) +return iommu_sva_unbind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data->hpasid); +else +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); }/**
drivers/vfio/vfio_iommu_type1.c +0
-0@@ -2429,7 +2441,12 @@ static int vfio_cache_inv_fn(struct device *dev, void *data) struct iommu_cache_invalidate_info *cache_inv_info =
drivers/vfio/vfio_iommu_type1.c static int vfio_cache_inv_fn(struct device *dev, · line 2451
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +6
-1(struct iommu_cache_invalidate_info *) dc->data; -return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +if (dc->group->mdev_group) +return iommu_cache_invalidate (dc->domain, +vfio_mdev_get_iommu_device (dev), cache_inv_info); +else +return iommu_cache_invalidate (dc->domain, + dev, cache_inv_info); }
possibly above could be simplified, e.g.
static struct device *vfio_get_iommu_device(struct vfio_group *group,
struct device *dev)
{
if (group->mdev_group)
return vfio_mdev_get_iommu_device(dev);
else
return dev;
}Then use it to replace plain 'dev' in all three places.
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +0
-1static long vfio_iommu_type1_ioctl(void *iommu_data, --2.7.4
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 8/8] vfio/type1: Add vSVA support for IOMMU-backed mdevs
Quoted 9 lines
From: Tian, Kevin <[email protected]>
Sent: Monday, March 30, 2020 9:19 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 8/8] vfio/type1: Add vSVA support for IOMMU-backed mdevsQuoted 3 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PM
Quoted 6 lines
Quoted 4 lines
From: Liu Yi L <[email protected]>
Recent years, mediated device pass-through framework (e.g. vfio-mdev) are used to achieve flexible device sharing across domains (e.g. VMs).
are->is
got it.
Quoted 11 lines
Quoted 8 lines
Also there are hardware assisted mediated pass-through solutions from platform vendors. e.g. Intel VT-d scalable mode which supports Intel Scalable I/O Virtualization technology. Such mdevs are called IOMMU- backed mdevs as there are IOMMU enforced DMA isolation for such mdevs. In kernel, IOMMU-backed mdevs are exposed to IOMMU layer by aux- domain concept, which means mdevs are protected by an iommu domain which is aux-domain of its physical device. Details can be found in the KVM
"by an iommu domain which is auxiliary to the domain that the kernel driver primarily uses for DMA API"
yep.
Quoted 5 lines
Quoted 5 lines
presentation from Kevin Tian. IOMMU-backed equals to IOMMU-capable.
https://events19.linuxfoundation.org/wp-content/uploads/2017/12/\
Hardware-Assisted-Mediated-Pass-Through-with-VFIO-Kevin-Tian-Intel.pdf
Quoted 7 lines
Quoted 4 lines
This patch supports NESTING IOMMU for IOMMU-backed mdevs by figuring out the physical device of an IOMMU-backed mdev and then invoking IOMMU requests to IOMMU layer with the physical device and the mdev's aux domain info.
"and then calling into the IOMMU layer to complete the vSVA operations on the aux domain associated with that mdev"
got it.
Quoted 85 lines
Quoted 70 lines
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used on IOMMU-backed mdevs.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>CC Jun Tian <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)(quoted diff) +0
-0
b/drivers/vfio/vfio_iommu_type1.cindex 937ec3f..d473665 100644drivers/vfio/vfio_iommu_type1.c +2
-0@@ -132,6 +132,7 @@ struct vfio_regions {struct domain_capsule {struct iommu_domain *domain; +struct vfio_group *group;void *data; };@@ -148,6 +149,7 @@ static int vfio_iommu_for_each_dev(struct vfio_iommu *iommu, list_for_each_entry(d, &iommu->domain_list, next) { dc.domain = d->domain;list_for_each_entry (g, &d->group_list, next) { + dc.group = g; ret =iommu_group_for_each_dev (g->iommu_group, &dc, fn);drivers/vfio/vfio_iommu_type1.c +6
-1if (ret)@@ -2347,7 +2349,12 @@ static int vfio_bind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +if (dc->group->mdev_group) +return iommu_sva_bind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data); +else +return iommu_sva_bind_gpasid (dc->domain, + dev, gbind_data); }static int
vfio_unbind_gpasid_fn(struct device *dev, void *data)@@
-2356,8 +2363,13 @@ static intvfio_unbind_gpasid_fn(struct device *dev, void *data)drivers/vfio/vfio_iommu_type1.c +6
-1struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_unbind_gpasid (dc->domain, dev, +if (dc->group->mdev_group) +return iommu_sva_unbind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data->hpasid); +else +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); }/** @@ -2429,7 +2441,12 @@ static int vfio_cache_inv_fn(struct device *dev, void *data) struct iommu_cache_invalidate_info *cache_inv_info =drivers/vfio/vfio_iommu_type1.c static int vfio_cache_inv_fn(struct device · line 2451
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +6
-1(struct iommu_cache_invalidate_info *) dc->data; -return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +if (dc->group->mdev_group) +return iommu_cache_invalidate (dc->domain, +vfio_mdev_get_iommu_device (dev), cache_inv_info); +else +return iommu_cache_invalidate (dc->domain, + dev, cache_inv_info); }possibly above could be simplified, e.g.
static struct device *vfio_get_iommu_device(struct
vfio_group*group,
struct device *dev)
Quoted 8 lines
{ if (group->mdev_group) return vfio_mdev_get_iommu_device(dev); else return dev; }Then use it to replace plain 'dev' in all three places.
yes, better for reading. thanks.
Regards,
Yi Liu
Re: [PATCH v1 8/8] vfio/type1: Add vSVA support for IOMMU-backed mdevs
In reply to Liu, Yi LQuoted 56 lines · Yi L
From: Liu Yi L <[email protected]>
Recent years, mediated device pass-through framework (e.g. vfio-mdev) are used to achieve flexible device sharing across domains (e.g. VMs). Also there are hardware assisted mediated pass-through solutions from platform vendors. e.g. Intel VT-d scalable mode which supports Intel Scalable I/O Virtualization technology. Such mdevs are called IOMMU- backed mdevs as there are IOMMU enforced DMA isolation for such mdevs. In kernel, IOMMU-backed mdevs are exposed to IOMMU layer by aux-domain concept, which means mdevs are protected by an iommu domain which is aux-domain of its physical device. Details can be found in the KVM presentation from Kevin Tian. IOMMU-backed equals to IOMMU-capable.
https://events19.linuxfoundation.org/wp-content/uploads/2017/12/\
Hardware-Assisted-Mediated-Pass-Through-with-VFIO-Kevin-Tian-Intel.pdfThis patch supports NESTING IOMMU for IOMMU-backed mdevs by figuring out the physical device of an IOMMU-backed mdev and then invoking IOMMU requests to IOMMU layer with the physical device and the mdev's aux domain info.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used on IOMMU-backed mdevs.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>CC Jun Tian <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)drivers/vfio/vfio_iommu_type1.c +2
-0@@ -132,6 +132,7 @@ struct vfio_regions {struct domain_capsule {struct iommu_domain *domain; +struct vfio_group *group;void *data; };@@ -148,6 +149,7 @@ static int vfio_iommu_for_each_dev(struct vfio_iommu *iommu,list_for_each_entry (d, &iommu->domain_list, next) { dc.domain = d->domain;list_for_each_entry (g, &d->group_list, next) { + dc.group = g; ret =iommu_group_for_each_dev (g->iommu_group, &dc, fn);if (ret)
drivers/vfio/vfio_iommu_type1.c static int vfio_bind_gpasid_fn(struct device *dev, void *data) · line 2354
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-1@@ -2347,7 +2349,12 @@ static int vfio_bind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +if (dc->group->mdev_group) +return iommu_sva_bind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data);
But we can't assume an mdev device is iommu backed, so this can call with NULL dev, which appears will pretty quickly segfault intel_svm_bind_gpasid.
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+else +return iommu_sva_bind_gpasid (dc->domain, + dev, gbind_data); }static int vfio_unbind_gpasid_fn (struct device *dev,void *data)
drivers/vfio/vfio_iommu_type1.c static int vfio_unbind_gpasid_fn(struct device *dev, void *data) · line 2369
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +3
-1@@ -2356,8 +2363,13 @@ static int vfio_unbind_gpasid_fn(struct device *dev, void *data)struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_unbind_gpasid (dc->domain, dev, +if (dc->group->mdev_group) +return iommu_sva_unbind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data->hpasid);
Same
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+else +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); }/**
drivers/vfio/vfio_iommu_type1.c static int vfio_cache_inv_fn(struct device *dev, void *data) · line 2446
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-1@@ -2429,7 +2441,12 @@ static int vfio_cache_inv_fn(struct device *dev, void *data)struct iommu_cache_invalidate_info *cache_inv_info = (struct iommu_cache_invalidate_info *) dc->data; -return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +if (dc->group->mdev_group) +return iommu_cache_invalidate (dc->domain, +vfio_mdev_get_iommu_device (dev), cache_inv_info);
And again
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+else +return iommu_cache_invalidate (dc->domain, + dev, cache_inv_info); } static long vfio_iommu_type1_ioctl(void *iommu_data,
RE: [PATCH v1 8/8] vfio/type1: Add vSVA support for IOMMU-backed mdevs
Hi Alex,
Quoted 70 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 4:34 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 8/8] vfio/type1: Add vSVA support for IOMMU-backed mdevsdrivers/vfio/vfio_iommu_type1.c static int vfio_iommu_for_each_dev(struct vfio_iommu
Quoted 48 lines · Yi L
From: Liu Yi L <[email protected]>
Recent years, mediated device pass-through framework (e.g. vfio-mdev) are used to achieve flexible device sharing across domains (e.g. VMs). Also there are hardware assisted mediated pass-through solutions from platform vendors. e.g. Intel VT-d scalable mode which supports Intel Scalable I/O Virtualization technology. Such mdevs are called IOMMU- backed mdevs as there are IOMMU enforced DMA isolation for such mdevs. In kernel, IOMMU-backed mdevs are exposed to IOMMU layer by aux-domain concept, which means mdevs are protected by an iommu domain which is aux-domain of its physical device. Details can be found in the KVM presentation from Kevin Tian. IOMMU-backed equals to IOMMU-capable.
https://events19.linuxfoundation.org/wp-content/uploads/2017/12/\
Hardware-Assisted-Mediated-Pass-Through-with-VFIO-Kevin-Tian-Intel.pdfThis patch supports NESTING IOMMU for IOMMU-backed mdevs by figuring out the physical device of an IOMMU-backed mdev and then invoking IOMMU requests to IOMMU layer with the physical device and the mdev's aux domain info.
With this patch, vSVA (Virtual Shared Virtual Addressing) can be used on IOMMU-backed mdevs.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>CC Jun Tian <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)drivers/vfio/vfio_iommu_type1.c +1
-0@@ -132,6 +132,7 @@ struct vfio_regions {struct domain_capsule {struct iommu_domain *domain; +struct vfio_group *group;void *data; };@@ -148,6 +149,7 @@ static int vfio_iommu_for_each_dev(struct vfio_iommu*iommu,
drivers/vfio/vfio_iommu_type1.c static int vfio_bind_gpasid_fn(struct device *dev, void
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +1
-0list_for_each_entry (d, &iommu->domain_list, next) { dc.domain = d->domain;list_for_each_entry (g, &d->group_list, next) { + dc.group = g; ret =iommu_group_for_each_dev (g->iommu_group, &dc, fn);if (ret)@@ -2347,7 +2349,12 @@ static int vfio_bind_gpasid_fn(struct device *dev, void*data)
Quoted 3 lines
drivers/vfio/vfio_iommu_type1.c +0
-0struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data;
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_bind_gpasid_fn(struct device *dev, void *data) · line 2354
Quoted 4 lines
drivers/vfio/vfio_iommu_type1.c +3
-1-return iommu_sva_bind_gpasid (dc->domain, dev, gbind_data); +if (dc->group->mdev_group) +return iommu_sva_bind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data);But we can't assume an mdev device is iommu backed, so this can call with NULL dev, which appears will pretty quickly segfault
intel_svm_bind_gpasid.
I don't think the non-iommu backed mdev will not be in the iommu->domain_list. right? But, yeah, from this function p.o.v , it is still necessary to do a check. How about adding a check on the return of vfio_mdev_get_iommu_device(dev)? If iommu_device is fetch, the mdev should be iommu-backed. does it make sense?
Regards,
Yi Liu
Quoted 44 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_unbind_gpasid_fn(struct device *dev,
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+else +return iommu_sva_bind_gpasid (dc->domain, + dev, gbind_data); }static int vfio_unbind_gpasid_fn (struct device *dev,void *data)@@ -2356,8 +2363,13 @@ static int vfio_unbind_gpasid_fn(struct device *dev,void *data)
drivers/vfio/vfio_iommu_type1.c static int vfio_unbind_gpasid_fn(struct device *dev, void *data) · line 2369
Quoted 8 lines
drivers/vfio/vfio_iommu_type1.c +3
-1struct iommu_gpasid_bind_data *gbind_data = (struct iommu_gpasid_bind_data *) dc->data; -return iommu_sva_unbind_gpasid (dc->domain, dev, +if (dc->group->mdev_group) +return iommu_sva_unbind_gpasid (dc->domain, +vfio_mdev_get_iommu_device (dev), gbind_data->hpasid);Same
drivers/vfio/vfio_iommu_type1.c static int vfio_cache_inv_fn(struct device *dev, void
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+else +return iommu_sva_unbind_gpasid (dc->domain, dev, + gbind_data->hpasid); }/** @@ -2429,7 +2441,12 @@ static int vfio_cache_inv_fn(struct device *dev, void*data)
drivers/vfio/vfio_iommu_type1.c static int vfio_cache_inv_fn(struct device *dev, void *data) · line 2446
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +3
-1struct iommu_cache_invalidate_info *cache_inv_info = (struct iommu_cache_invalidate_info *) dc->data; -return iommu_cache_invalidate (dc->domain, dev, cache_inv_info); +if (dc->group->mdev_group) +return iommu_cache_invalidate (dc->domain, +vfio_mdev_get_iommu_device (dev), cache_inv_info);And again
Quoted 6 lines
drivers/vfio/vfio_iommu_type1.c +3
-0+else +return iommu_cache_invalidate (dc->domain, + dev, cache_inv_info); } static long vfio_iommu_type1_ioctl(void *iommu_data,
Quoted 66 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:32 PM
To: [email protected]; [email protected]
Subject: [PATCH v1 0/8] vfio: expose virtual Shared Virtual Addressing to VMsFrom: Liu Yi L <[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
This VFIO series is intended to expose SVA usage to VMs. i.e. Sharing guest application address space with passthru devices. This is called vSVA in this series. The whole vSVA enabling requires QEMU/VFIO/IOMMU changes. For IOMMU and QEMU changes, they are in separate series (listed in the "Related series").
The high-level architecture for SVA virtualization is as below, the key design of vSVA support is to utilize the dual-stage IOMMU translation ( also known as IOMMU nesting translation) capability in host IOMMU.
.-------------. .---------------------------. | vIOMMU | | Guest process CR3, FL only| | | '---------------------------'.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest------| Shadow |--------------------------|-------- v v v Host .-------------. .----------------------. | pIOMMU | | Bind FL for GVA-GPA | | | '----------------------' .----------------/ | | PASID Entry | V (Nested xlate) '----------------\.------------------------------. | | |SL for GPA-HPA, default domain| | | '------------------------------' '-------------'Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tablesThere are roughly four parts in this patchset which are corresponding to the basic vSVA support for PCI device assignment
1. vfio support for PASID allocation and free for VMs
2. vfio support for guest page table binding request from VMs
3. vfio support for IOMMU cache invalidation from VMs
4. vfio support for vSVA usage on IOMMU-backed mdevsThe complete vSVA kernel upstream patches are divided into three phases:
1. Common APIs and PCI device direct assignment
2. IOMMU-backed Mediated Device assignment
3. Page Request Services (PRS) supportThis patchset is aiming for the phase 1 and phase 2, and based on Jacob's below series.
[PATCH V10 00/11] Nested Shared Virtual Address (SVA) VT-d support:
https://lkml.org/lkml/2020/3/20/1172
Quoted 8 lines
Complete set for current vSVA can be found in below branch.
https://github.com/luxis1999/linux-vsva.git: vsva-linux-5.6-rc6The corresponding QEMU patch series is as below, complete QEMU set can be found in below branch.
[PATCH v1 00/22]intel_iommu: expose Shared Virtual Addressing to VMs complete QEMU set can be found in below link:
https://github.com/luxis1999/qemu.git:sva_vtd_v10_v1
The ioasid extension is in the below link.
https://lkml.org/lkml/2020/3/25/874
Regards,
Yi Liu