Add multiple address spaces support to VDUSE
Eugenio Pérez
maintainerv9 cover Add multiple address spaces support to VDUSE
When used by vhost-vDPA bus driver for VM, the control virtqueue should be shadowed via userspace VMM (QEMU) instead of being assigned directly to Guest. This is because QEMU needs to know the device state in order to start and stop device correctly (e.g for Live Migration).
This requies to isolate the memory mapping for control virtqueue presented by vhost-vDPA to prevent guest from accessing it directly.
This series add support to multiple address spaces in VDUSE device allowing selective virtqueue isolation through address space IDs (ASID).
The VDUSE device needs to report:
* Number of virtqueue groups
* Association of each vq group with each virtqueue
* Number of address spaces supported.
Then, the vDPA driver can modify the ASID assigned to each VQ group to isolate the memory AS. This aligns VDUSE with vdpa_sim and nvidia mlx5 devices which already support ASID.
This helps to isolate the environments for the virtqueues that will not be assigned directly. E.g in the case of virtio-net, the control virtqueue will not be assigned directly to guest.
Also, to be able to test this patch, the user needs to manually revert 56e71885b034 ("vduse: Temporarily fail if control queue feature requested").
Tested by creating a VDUSE device OVS with and without MQ, and live migrating between two hosts back and forth while maintaining ping alive in all the stages. All tested with and without lockdep.
PATCH v9:
* Change to RCU.
PATCH v8:
* Revert the change from mutex to rwlock (MST).
PATCH v7:
* Fix not taking the write lock in the registering vdpa device error path (Jason).
PATCH v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ in group and nas handling at device creation (MST).
* Fix struct name not matching in the doc.
* s/sepparate/separate (MST).
PATCH v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Revert core vdpa changes (Jason).
* Fix group == ngroup case in checking VQ_SETUP argument (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.
PATCH v4:
* Consider config->nas == 0 and config->ngroups == 0 as a fail (Jason).
* Revert the "invalid vq group" concept and assume 0 if not set.
* Divide each domain bounce size between the device bounce size (Jason).
* Revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
PATCH v3:
* Make the default group an invalid group as long as VDUSE device does not set it to some valid u32 value. Modify the vdpa core to take that into account (Jason). Adapt all the virtio_map_ops callbacks to it.
* Make setting status DRIVER_OK fail if vq group is not valid.
* Create the VDUSE_DEV_MAX_GROUPS and VDUSE_DEV_MAX_AS instead of using a magic number
* Remove the _int name suffix from struct vduse_vq_group.
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash the patch implementing the AS logic with the patch creating the vduse_as struct (Jason).
PATCH v2:
* Now the vq group is in vduse_vq_config struct instead of issuing one VDUSE message per vq.
* Convert the use of mutex to rwlock (Xie Yongji).
PATCH v1:
* Fix: Remove BIT_ULL(VIRTIO_S_*), as _S_ is already the bit (Maxime)
* Using vduse_vq_group_int directly instead of an empty struct in union virtio_map.
RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs. Also, it's a dynamic array now. Same with ASID.
* Move the valid vq groups range check to vduse_validate_config.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Use of array_index_nospec in VDUSE device ioctls.
* Move the umem mutex to asid struct so there is no contention between ASIDs.
* Remove the descs vq group capability as it will not be used and we can add it on top.
* Do not ask for vq groups in number of vq groups < 2.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
RFC v2:
* Cache group information in kernel, as we need to provide the vq map tokens properly.
* Add descs vq group to optimize SVQ forwarding and support indirect descriptors out of the box.
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.
* Fixes detected testing with OVS+VDUSE.
Eugenio Pérez (6):
vduse: add v1 API definition
vduse: add vq group support
vduse: return internal vq group struct as map token
vduse: refactor vdpa_dev_add for goto err handling
vduse: add vq group asid support
vduse: bump version number
drivers/vdpa/vdpa_user/vduse_dev.c | 496 +++++++++++++++++++++++------ include/linux/virtio.h | 6 +- include/uapi/linux/vduse.h | 67 +++- 3 files changed, 455 insertions(+), 114 deletions(-)
Signature · 1 line
2.51.1
Eugenio Pérez
maintainerPATCH v9 1/6 vduse: add v1 API definition
This allows the kernel to detect whether the userspace VDUSE device supports the VQ group and ASID features. VDUSE devices that don't set the V1 API will not receive the new messages, and vdpa device will be created with only one vq group and asid.
The next patches implement the new feature incrementally, only enabling the VDUSE device to set the V1 API version by the end of the series.
--- include/uapi/linux/vduse.h | 4 ++++ 1 file changed, 4 insertions(+)
include/uapi/linux/vduse.h +4 -0
@@ -10,6 +10,10 @@#define VDUSE_API_VERSION 0 +/* VQ groups and ASID support */ + +#define VDUSE_API_VERSION_1 1 +/* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension.
Signature · 1 line
2.51.1
Eugenio Pérez
maintainerPATCH v9 2/6 vduse: add vq group support
In reply to Eugenio PérezThis allows separate the different virtqueues in groups that shares the same address space. Asking the VDUSE device for the groups of the vq at the beginning as they're needed for the DMA API.
Allocating 3 vq groups as net is the device that need the most groups:
* Dataplane (guest passthrough)
* CVQ
* Shadowed vrings.
Future versions of the series can include dynamic allocation of the groups array so VDUSE can declare more groups.
---
v6:
* s/sepparate/separate (MST).
* s/dev->api_version < 1/dev->api_version < VDUSE_API_VERSION_1
v5:
* Revert core vdpa changes (Jason).
* Fix group == ngroup case in checking VQ_SETUP argument (Jason).
v4:
* Revert the "invalid vq group" concept and assume 0 if not set (Jason).
* Make config->ngroups == 0 invalid (Jason).
v3:
* Make the default group an invalid group as long as VDUSE device does not set it to some valid u32 value. Modify the vdpa core to take that into account (Jason).
* Create the VDUSE_DEV_MAX_GROUPS instead of using a magic number
v2:
* Now the vq group is in vduse_vq_config struct instead of issuing one VDUSE message per vq.
v1:
* Fix: Remove BIT_ULL(VIRTIO_S_*), as _S_ is already the bit (Maxime)
RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove the descs vq group capability as it will not be used and we can add it on top.
* Do not ask for vq groups in number of vq groups < 2.
* Move the valid vq groups range check to vduse_validate_config.
RFC v2:
* Cache group information in kernel, as we need to provide the vq map tokens properly.
* Add descs vq group to optimize SVQ forwarding and support indirect descriptors out of the box.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 48 ++++++++++++++++++++++++++---- include/uapi/linux/vduse.h | 12 ++++++-- 2 files changed, 52 insertions(+), 8 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +43 -5
@@ -39,6 +39,7 @@#define DRV_LICENSE "GPL v2"#define VDUSE_DEV_MAX (1U << MINORBITS) +#define VDUSE_DEV_MAX_GROUPS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -58,6 +59,7 @@ struct vduse_virtqueue {struct vdpa_vq_state state;bool ready;bool kicked; +u32 vq_group;spinlock_t kick_lock;spinlock_t irq_lock;struct eventfd_ctx *kickfd;
@@ -114,6 +116,7 @@ struct vduse_dev {u8 status;u32 vq_num;u32 vq_align; +u32 ngroups;struct vduse_umem *umem;struct mutex mem_lock;unsigned int bounce_size;
@@ -455,6 +458,7 @@ static void vduse_dev_reset(struct vduse_dev *dev) vq->driver_addr =0 ; vq->device_addr =0 ; vq->num =0 ; + vq->vq_group =0 ;memset (&vq->state,0 ,sizeof (vq->state));spin_lock (&vq->kick_lock);
@@ -592,6 +596,16 @@ static int vduse_vdpa_set_vq_state(struct vdpa_device *vdpa, u16 idx,return 0 ; } +static u32 vduse_get_vq_group (struct vdpa_device *vdpa,u16 idx) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); + +if (dev->api_version <VDUSE_API_VERSION_1 ) +return 0 ; + +return dev->vqs[idx]->vq_group; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {
@@ -789,6 +803,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .set_vq_cb = vduse_vdpa_set_vq_cb, .set_vq_num = vduse_vdpa_set_vq_num, .get_vq_size = vduse_vdpa_get_vq_size, + .get_vq_group = vduse_get_vq_group, .set_vq_ready = vduse_vdpa_set_vq_ready, .get_vq_ready = vduse_vdpa_get_vq_ready, .set_vq_state = vduse_vdpa_set_vq_state,
@@ -1252,12 +1267,24 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,if (config.index >= dev->vq_num)break ; -if (!is_mem_zero ((const char *)config.reserved, -sizeof (config.reserved))) +if (dev->api_version <VDUSE_API_VERSION_1 && config.group) +break ; + +if (dev->api_version >=VDUSE_API_VERSION_1 ) { +if (config.group >= dev->ngroups) +break ; +if (dev->status &VIRTIO_CONFIG_S_DRIVER_OK ) +break ; + } + +if (config.reserved1 || + !is_mem_zero ((const char *)config.reserved2, +sizeof (config.reserved2)))break ; index =array_index_nospec (config.index, dev->vq_num); dev->vqs[index]->num_max = config.max_size; + dev->vqs[index]->vq_group = config.group; ret =0 ;break ; }
@@ -1737,12 +1764,20 @@ static bool features_is_valid(struct vduse_dev_config *config)return true; } -static bool vduse_validate_config (struct vduse_dev_config *config) +static bool vduse_validate_config (struct vduse_dev_config *config, +u64 api_version) {if (!is_mem_zero ((const char *)config->reserved,sizeof (config->reserved)))return false; +if (api_version <VDUSE_API_VERSION_1 && config->ngroups) +return false; + +if (api_version >=VDUSE_API_VERSION_1 && + (!config->ngroups || config->ngroups >VDUSE_DEV_MAX_GROUPS )) +return false; +if (config->vq_align >PAGE_SIZE )return false;
@@ -1858,6 +1893,9 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->device_features = config->features; dev->device_id = config->device_id; dev->vendor_id = config->vendor_id; + dev->ngroups = (dev->api_version <VDUSE_API_VERSION_1 ) + ?1 + : config->ngroups; dev->name =kstrdup (config->name,GFP_KERNEL );if (!dev->name) goto err_str;
@@ -1936,7 +1974,7 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,break ; ret = -EINVAL ; -if (vduse_validate_config (&config) == false) +if (!vduse_validate_config (&config, control->api_version))break ; buf =vmemdup_user (argp + size, config.config_size);
@@ -2017,7 +2055,7 @@ static int vduse_dev_init_vdpa(struct vduse_dev *dev, const char *name) vdev = vdpa_alloc_device(struct vduse_vdpa, vdpa, dev->dev, &vduse_vdpa_config_ops, &vduse_map_ops, -1 ,1 , name, true); + dev->ngroups,1 , name, true);if (IS_ERR (vdev))return PTR_ERR (vdev);
include/uapi/linux/vduse.h +9 -3
@@ -31,6 +31,7 @@* @features: virtio features * @vq_num: the number of virtqueues * @vq_align: the allocation alignment of virtqueue's metadata +* @ngroups: number of vq groups that VDUSE device declares * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space
@@ -45,7 +46,8 @@ struct vduse_dev_config {__u64 features;__u32 vq_num;__u32 vq_align; -__u32 reserved[13 ]; +__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ +__u32 reserved[12 ];__u32 config_size;__u8 config[]; };
@@ -122,14 +124,18 @@ struct vduse_config_data {* struct vduse_vq_config - basic configuration of a virtqueue * @index: virtqueue index * @max_size: the max size of virtqueue -* @reserved: for future use, needs to be initialized to zero +* @reserved1: for future use, needs to be initialized to zero +* @group: virtqueue group +* @reserved2: for future use, needs to be initialized to zero * * Structure used by VDUSE_VQ_SETUP ioctl to setup a virtqueue. */ struct vduse_vq_config {__u32 index;__u16 max_size; -__u16 reserved[13 ]; +__u16 reserved1; +__u32 group; +__u16 reserved2[10 ]; };/*
Signature · 1 line
2.51.1
Eugenio Pérez
maintainerPATCH v9 3/6 vduse: return internal vq group struct as map token
In reply to Eugenio PérezReturn the internal struct that represents the vq group as virtqueue map token, instead of the device. This allows the map functions to access the information per group.
At this moment all the virtqueues share the same vq group, that only can point to ASID 0. This change prepares the infrastructure for actual per-group address space handling
---
v4:
* Revert the "invalid vq group" concept, and assume 0 by default.
* Revert unnecesary blank line addition (Jason)
v3:
* Adapt all virtio_map_ops callbacks to handle empty tokens in case of invalid groups.
* Make setting status DRIVER_OK fail if vq group is not valid.
* Remove the _int name suffix from struct vduse_vq_group.
RFC v3:
* Make the vq groups a dynamic array to support an arbitrary number of them.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 100 ++++++++++++++++++++++++++--- include/linux/virtio.h | 6 +- 2 files changed, 94 insertions(+), 12 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +91 -9
@@ -22,6 +22,7 @@#include <linux/uio.h> #include <linux/vdpa.h> #include <linux/nospec.h> +#include <linux/virtio.h> #include <linux/vmalloc.h> #include <linux/sched/mm.h> #include <uapi/linux/vduse.h>
@@ -85,6 +86,10 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_vq_group { +struct vduse_dev *dev; +}; +struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;
@@ -118,6 +123,7 @@ struct vduse_dev {u32 vq_align;u32 ngroups;struct vduse_umem *umem; +struct vduse_vq_group *groups;struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock;
@@ -606,6 +612,17 @@ static u32 vduse_get_vq_group(struct vdpa_device *vdpa, u16 idx)return dev->vqs[idx]->vq_group; } +static union virtio_map vduse_get_vq_map (struct vdpa_device *vdpa,u16 idx) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +u32 vq_group =vduse_get_vq_group (vdpa, idx); +union virtio_map ret = { + .group = &dev->groups[vq_group], + }; + +return ret; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {
@@ -826,6 +843,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };
@@ -833,7 +851,14 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t dma_addr, size_t size, enum dma_data_direction dir) { - struct vduse_iova_domain *domain = token.iova_domain; + struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return ; + + vdev = token.group->dev; + domain = vdev->domain;vduse_domain_sync_single_for_device (domain, dma_addr, size, dir); }
@@ -842,7 +867,14 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, dma_addr_t dma_addr, size_t size, enum dma_data_direction dir) { - struct vduse_iova_domain *domain = token.iova_domain; + struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return ; + + vdev = token.group->dev; + domain = vdev->domain;vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }
@@ -852,7 +884,14 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, enum dma_data_direction dir, unsigned long attrs) { -struct vduse_iova_domain *domain = token.iova_domain; +struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return DMA_MAPPING_ERROR ; + + vdev = token.group->dev; + domain = vdev->domain;return vduse_domain_map_page (domain, page, offset, size, dir, attrs); }
@@ -861,7 +900,14 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr, size_t size, enum dma_data_direction dir, unsigned long attrs) { - struct vduse_iova_domain *domain = token.iova_domain; + struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return ; + + vdev = token.group->dev; + domain = vdev->domain;return vduse_domain_unmap_page (domain, dma_addr, size, dir, attrs); }
@@ -869,11 +915,17 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,static void *vduse_dev_alloc_coherent (union virtio_map token,size_t size,dma_addr_t *dma_addr,gfp_t flag) { -struct vduse_iova_domain *domain = token.iova_domain; +struct vduse_dev *vdev; +struct vduse_iova_domain *domain;unsigned long iova;void *addr; *dma_addr =DMA_MAPPING_ERROR ; +if (!token.group) +return NULL ; + + vdev = token.group->dev; + domain = vdev->domain; addr =vduse_domain_alloc_coherent (domain, size, (dma_addr_t *)&iova, flag);if (!addr)
@@ -888,14 +940,28 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size, void *vaddr, dma_addr_t dma_addr, unsigned long attrs) { - struct vduse_iova_domain *domain = token.iova_domain; + struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return ; + + vdev = token.group->dev; + domain = vdev->domain;vduse_domain_free_coherent (domain, size, vaddr, dma_addr, attrs); }static bool vduse_dev_need_sync (union virtio_map token,dma_addr_t dma_addr) { -struct vduse_iova_domain *domain = token.iova_domain; +struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return false; + + vdev = token.group->dev; + domain = vdev->domain;return dma_addr < domain->bounce_size; }
@@ -909,7 +975,14 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)static size_t vduse_dev_max_mapping_size (union virtio_map token) { -struct vduse_iova_domain *domain = token.iova_domain; +struct vduse_dev *vdev; +struct vduse_iova_domain *domain; + +if (!token.group) +return 0 ; + + vdev = token.group->dev; + domain = vdev->domain;return domain->bounce_size; }
@@ -1727,6 +1800,7 @@ static int vduse_destroy_dev(char *name)if (dev->domain)vduse_domain_destroy (dev->domain);kfree (dev->name); +kfree (dev->groups);vduse_dev_destroy (dev);module_put (THIS_MODULE );
@@ -1896,6 +1970,13 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->ngroups = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->ngroups; + dev->groups =kcalloc (dev->ngroups,sizeof (dev->groups[0 ]), +GFP_KERNEL ); +if (!dev->groups) + goto err_vq_groups; +for (u32 i =0 ; i < dev->ngroups; ++i) + dev->groups[i].dev = dev; + dev->name =kstrdup (config->name,GFP_KERNEL );if (!dev->name) goto err_str;
@@ -1932,6 +2013,8 @@ static int vduse_create_dev(struct vduse_dev_config *config, err_idr:kfree (dev->name); err_str: +kfree (dev->groups); +err_vq_groups:vduse_dev_destroy (dev); err:return ret;
@@ -2093,7 +2176,6 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,return -ENOMEM ; } - dev->vdev->vdpa.vmap.iova_domain = dev->domain; ret =_vdpa_register_device (&dev->vdev->vdpa, dev->vq_num);if (ret) {put_device (&dev->vdev->vdpa.dev);
include/linux/virtio.h +3 -3
@@ -41,13 +41,13 @@ struct virtqueue {void *priv; }; -struct vduse_iova_domain ; +struct vduse_vq_group ;union virtio_map {/* Device that performs DMA */ struct device *dma_dev; -/* VDUSE specific mapping data */ -struct vduse_iova_domain *iova_domain; +/* VDUSE specific virtqueue group for doing map */ +struct vduse_vq_group *group; }; int virtqueue_add_outbuf(struct virtqueue *vq,
Signature · 1 line
2.51.1
Eugenio Pérez
maintainerPATCH v9 4/6 vduse: refactor vdpa_dev_add for goto err handling
In reply to Eugenio PérezNext patches introduce more error paths in this function. Refactor it so they can be accomodated through gotos.
---
v6: New in v6.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +14 -8
@@ -2172,21 +2172,27 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name, dev->bounce_size);mutex_unlock (&dev->domain_lock);if (!dev->domain) { -put_device (&dev->vdev->vdpa.dev); -return -ENOMEM ; + ret = -ENOMEM ; + goto domain_err; } ret =_vdpa_register_device (&dev->vdev->vdpa, dev->vq_num);if (ret) { -put_device (&dev->vdev->vdpa.dev); -mutex_lock (&dev->domain_lock); -vduse_domain_destroy (dev->domain); - dev->domain =NULL ; -mutex_unlock (&dev->domain_lock); -return ret; + goto register_err; }return 0 ; + +register_err: +mutex_lock (&dev->domain_lock); +vduse_domain_destroy (dev->domain); + dev->domain =NULL ; +mutex_unlock (&dev->domain_lock); + +domain_err: +put_device (&dev->vdev->vdpa.dev); + +return ret; }static void vdpa_dev_del (struct vdpa_mgmt_dev *mdev,struct vdpa_device *dev)
Signature · 1 line
2.51.1
Eugenio Pérez
maintainerPATCH v9 5/6 vduse: add vq group asid support
In reply to Eugenio PérezAdd support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.
---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.
v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.
v7:
* Take write lock in the error path (Jason).
v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.
v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.
v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).
v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)
v2:
* Convert the use of mutex to rwlock.
RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.
RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +264 -106
@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h>
@@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };
@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; } +/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]); +} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir); }
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs); -return vduse_domain_map_page (domain, page, offset, size, dir, attrs); +return r; } static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
@@ -901,21 +969,25 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - -return vduse_domain_unmap_page (domain, dma_addr, size, dir, attrs); +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); +vduse_domain_unmap_page (domain, dma_addr, size, dir, attrs); }static void *vduse_dev_alloc_coherent (union virtio_map token,size_t size,dma_addr_t *dma_addr,gfp_t flag) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;unsigned long iova;void *addr;
@@ -925,13 +997,14 @@ static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,return NULL ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); addr =vduse_domain_alloc_coherent (domain, size, (dma_addr_t *)&iova, flag); -if (!addr) -return NULL ; - - *dma_addr = (dma_addr_t )iova; +if (addr) + *dma_addr = (dma_addr_t )iova;return addr; }
@@ -941,29 +1014,39 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_free_coherent (domain, size, vaddr, dma_addr, attrs); +mutex_unlock (&vdev->domain_lock); }static bool vduse_dev_need_sync (union virtio_map token,dma_addr_t dma_addr) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +size_t bounce_size;if (!token.group)return false; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + bounce_size = domain->bounce_size; -return dma_addr < domain->bounce_size; +return dma_addr < bounce_size; }static int vduse_dev_mapping_error (union virtio_map token,dma_addr_t dma_addr)
@@ -976,15 +1059,22 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)static size_t vduse_dev_max_mapping_size (union virtio_map token) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +size_t bounce_size;if (!token.group)return 0 ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + domain = token.group->as->domain; + bounce_size = domain->bounce_size; -return domain->bounce_size; +return bounce_size; }static const struct virtio_map_ops vduse_map_ops = {
@@ -1124,39 +1214,40 @@ static int vduse_dev_queue_irq_work(struct vduse_dev *dev,return ret; } -static int vduse_dev_dereg_umem (struct vduse_dev *dev, +static int vduse_dev_dereg_umem (struct vduse_dev *dev,u32 asid,u64 iova,u64 size) {int ret; -mutex_lock (&dev->mem_lock); +mutex_lock (&dev->as[asid].mem_lock); ret = -ENOENT ; -if (!dev->umem) +if (!dev->as[asid].umem) goto unlock; ret = -EINVAL ; -if (!dev->domain) +if (!dev->as[asid].domain) goto unlock; -if (dev->umem->iova != iova || size != dev->domain->bounce_size) +if (dev->as[asid].umem->iova != iova || + size != dev->as[asid].domain->bounce_size) goto unlock; -vduse_domain_remove_user_bounce_pages (dev->domain); -unpin_user_pages_dirty_lock (dev->umem->pages, - dev->umem->npages, true); -atomic64_sub (dev->umem->npages, &dev->umem->mm->pinned_vm); -mmdrop (dev->umem->mm); -vfree (dev->umem->pages); -kfree (dev->umem); - dev->umem =NULL ; +vduse_domain_remove_user_bounce_pages (dev->as[asid].domain); +unpin_user_pages_dirty_lock (dev->as[asid].umem->pages, + dev->as[asid].umem->npages, true); +atomic64_sub (dev->as[asid].umem->npages, &dev->as[asid].umem->mm->pinned_vm); +mmdrop (dev->as[asid].umem->mm); +vfree (dev->as[asid].umem->pages); +kfree (dev->as[asid].umem); + dev->as[asid].umem =NULL ; ret =0 ; unlock: -mutex_unlock (&dev->mem_lock); +mutex_unlock (&dev->as[asid].mem_lock);return ret; }static int vduse_dev_reg_umem (struct vduse_dev *dev, -u64 iova,u64 uaddr,u64 size) +u32 asid,u64 iova,u64 uaddr,u64 size) {struct page **page_list =NULL ;struct vduse_umem *umem =NULL ;
@@ -1164,14 +1255,14 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev,unsigned long npages, lock_limit;int ret; -if (!dev->domain || !dev->domain->bounce_map || - size != dev->domain->bounce_size || +if (!dev->as[asid].domain || !dev->as[asid].domain->bounce_map || + size != dev->as[asid].domain->bounce_size || iova !=0 || uaddr & ~PAGE_MASK )return -EINVAL ; -mutex_lock (&dev->mem_lock); +mutex_lock (&dev->as[asid].mem_lock); ret = -EEXIST ; -if (dev->umem) +if (dev->as[asid].umem) goto unlock; ret = -ENOMEM ;
@@ -1195,7 +1286,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev, goto out; } - ret =vduse_domain_add_user_bounce_pages (dev->domain, + ret =vduse_domain_add_user_bounce_pages (dev->as[asid].domain, page_list, pinned);if (ret) goto out;
@@ -1208,7 +1299,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev, umem->mm = current->mm;mmgrab (current->mm); - dev->umem = umem; + dev->as[asid].umem = umem; out:if (ret && pinned >0 )unpin_user_pages (page_list, pinned);
@@ -1219,7 +1310,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev,vfree (page_list);kfree (umem); } -mutex_unlock (&dev->mem_lock); +mutex_unlock (&dev->as[asid].mem_lock);return ret; }
@@ -1251,47 +1342,66 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,switch (cmd) {case VDUSE_IOTLB_GET_FD : { -struct vduse_iotlb_entry entry; +struct vduse_iotlb_entry_v2 entry;struct vhost_iotlb_map *map;struct vdpa_map_file *map_file;struct file *f =NULL ; +u32 asid; ret = -EFAULT ; -if (copy_from_user (&entry, argp,sizeof (entry))) -break ; +if (dev->api_version >=VDUSE_API_VERSION_1 ) { +if (copy_from_user (&entry, argp,sizeof (entry))) +break ; + }else { + entry.asid =0 ; +if (copy_from_user (&entry.v1, argp, +sizeof (entry.v1))) +break ; + } ret = -EINVAL ; -if (entry.start > entry.last) +if (entry.v1.start > entry.v1.last) +break ; + +if (entry.asid >= dev->nas)break ;mutex_lock (&dev->domain_lock); -if (!dev->domain) { + asid =array_index_nospec (entry.asid, dev->nas); +if (!dev->as[asid].domain) {mutex_unlock (&dev->domain_lock);break ; } -spin_lock (&dev->domain->iotlb_lock); - map =vhost_iotlb_itree_first (dev->domain->iotlb, - entry.start, entry.last); +spin_lock (&dev->as[asid].domain->iotlb_lock); + map =vhost_iotlb_itree_first (dev->as[asid].domain->iotlb, + entry.v1.start, entry.v1.last);if (map) { map_file = (struct vdpa_map_file *)map->opaque; f =get_file (map_file->file); - entry.offset = map_file->offset; - entry.start = map->start; - entry.last = map->last; - entry.perm = map->perm; + entry.v1.offset = map_file->offset; + entry.v1.start = map->start; + entry.v1.last = map->last; + entry.v1.perm = map->perm; } -spin_unlock (&dev->domain->iotlb_lock); +spin_unlock (&dev->as[asid].domain->iotlb_lock);mutex_unlock (&dev->domain_lock); ret = -EINVAL ;if (!f)break ; - ret = -EFAULT ; -if (copy_to_user (argp, &entry,sizeof (entry))) { +if (dev->api_version >=VDUSE_API_VERSION_1 ) + ret =copy_to_user (argp, &entry, +sizeof (entry)); +else + ret =copy_to_user (argp, &entry.v1, +sizeof (entry.v1)); + +if (ret) { + ret = -EFAULT ;fput (f);break ; } - ret =receive_fd (f,NULL ,perm_to_file_flags (entry.perm)); + ret =receive_fd (f,NULL ,perm_to_file_flags (entry.v1.perm));fput (f);break ; }
@@ -1436,6 +1546,7 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd, }case VDUSE_IOTLB_REG_UMEM : {struct vduse_iova_umem umem; +u32 asid; ret = -EFAULT ;if (copy_from_user (&umem, argp,sizeof (umem)))
@@ -1443,17 +1554,21 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd, ret = -EINVAL ;if (!is_mem_zero ((const char *)umem.reserved, -sizeof (umem.reserved))) +sizeof (umem.reserved)) || + (dev->api_version <VDUSE_API_VERSION_1 && + umem.asid !=0 ) || umem.asid >= dev->nas)break ;mutex_lock (&dev->domain_lock); - ret =vduse_dev_reg_umem (dev, umem.iova, + asid =array_index_nospec (umem.asid, dev->nas); + ret =vduse_dev_reg_umem (dev, asid, umem.iova, umem.uaddr, umem.size);mutex_unlock (&dev->domain_lock);break ; }case VDUSE_IOTLB_DEREG_UMEM : {struct vduse_iova_umem umem; +u32 asid; ret = -EFAULT ;if (copy_from_user (&umem, argp,sizeof (umem)))
@@ -1461,10 +1576,15 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd, ret = -EINVAL ;if (!is_mem_zero ((const char *)umem.reserved, -sizeof (umem.reserved))) +sizeof (umem.reserved)) || + (dev->api_version <VDUSE_API_VERSION_1 && + umem.asid !=0 ) || + umem.asid >= dev->nas)break ; +mutex_lock (&dev->domain_lock); - ret =vduse_dev_dereg_umem (dev, umem.iova, + asid =array_index_nospec (umem.asid, dev->nas); + ret =vduse_dev_dereg_umem (dev, asid, umem.iova, umem.size);mutex_unlock (&dev->domain_lock);break ;
@@ -1472,6 +1592,7 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,case VDUSE_IOTLB_GET_INFO : {struct vduse_iova_info info;struct vhost_iotlb_map *map; +u32 asid; ret = -EFAULT ;if (copy_from_user (&info, argp,sizeof (info)))
@@ -1485,23 +1606,31 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd, sizeof(info.reserved)))break ; +if (dev->api_version <VDUSE_API_VERSION_1 ) { +if (info.asid) +break ; + }else if (info.asid >= dev->nas) +break ; +mutex_lock (&dev->domain_lock); -if (!dev->domain) { + asid =array_index_nospec (info.asid, dev->nas); +if (!dev->as[asid].domain) {mutex_unlock (&dev->domain_lock);break ; } -spin_lock (&dev->domain->iotlb_lock); - map =vhost_iotlb_itree_first (dev->domain->iotlb, +spin_lock (&dev->as[asid].domain->iotlb_lock); + map =vhost_iotlb_itree_first (dev->as[asid].domain->iotlb, info.start, info.last);if (map) { info.start = map->start; info.last = map->last; info.capability =0 ; -if (dev->domain->bounce_map && map->start ==0 && - map->last == dev->domain->bounce_size -1 ) +if (dev->as[asid].domain->bounce_map && + map->start ==0 && + map->last == dev->as[asid].domain->bounce_size -1 ) info.capability |=VDUSE_IOVA_CAP_UMEM ; } -spin_unlock (&dev->domain->iotlb_lock); +spin_unlock (&dev->as[asid].domain->iotlb_lock);mutex_unlock (&dev->domain_lock);if (!map)break ;
@@ -1526,8 +1655,10 @@ static int vduse_dev_release(struct inode *inode, struct file *file)struct vduse_dev *dev = file->private_data;mutex_lock (&dev->domain_lock); -if (dev->domain) -vduse_dev_dereg_umem (dev,0 , dev->domain->bounce_size); +for (int i =0 ; i < dev->nas; i++) +if (dev->as[i].domain) +vduse_dev_dereg_umem (dev, i,0 , + dev->as[i].domain->bounce_size);mutex_unlock (&dev->domain_lock);spin_lock (&dev->msg_lock);/* Make sure the inflight messages can processed after reconncection */
@@ -1746,7 +1877,6 @@ static struct vduse_dev *vduse_dev_create(void)return NULL ;mutex_init (&dev->lock); -mutex_init (&dev->mem_lock);mutex_init (&dev->domain_lock);spin_lock_init (&dev->msg_lock);INIT_LIST_HEAD (&dev->send_list);
@@ -1797,8 +1927,11 @@ static int vduse_destroy_dev(char *name)idr_remove (&vduse_idr, dev->minor);kvfree (dev->config);vduse_dev_deinit_vqs (dev); -if (dev->domain) -vduse_domain_destroy (dev->domain); +for (int i =0 ; i < dev->nas; i++) { +if (dev->as[i].domain) +vduse_domain_destroy (dev->as[i].domain); + } +kfree (dev->as);kfree (dev->name);kfree (dev->groups);vduse_dev_destroy (dev);
@@ -1845,12 +1978,17 @@ static bool vduse_validate_config(struct vduse_dev_config *config, sizeof(config->reserved)))return false; -if (api_version <VDUSE_API_VERSION_1 && config->ngroups) +if (api_version <VDUSE_API_VERSION_1 && + (config->ngroups || config->nas))return false; -if (api_version >=VDUSE_API_VERSION_1 && - (!config->ngroups || config->ngroups >VDUSE_DEV_MAX_GROUPS )) -return false; +if (api_version >=VDUSE_API_VERSION_1 ) { +if (!config->ngroups || config->ngroups >VDUSE_DEV_MAX_GROUPS ) +return false; + +if (!config->nas || config->nas >VDUSE_DEV_MAX_AS ) +return false; + }if (config->vq_align >PAGE_SIZE )return false;
@@ -1915,7 +2053,8 @@ static ssize_t bounce_size_store(struct device *device, ret = -EPERM ;mutex_lock (&dev->domain_lock); -if (dev->domain) +/* Assuming that if the first domain is allocated, all are allocated */ +if (dev->as[0 ].domain) goto unlock; ret =kstrtouint (buf,10 , &bounce_size);
@@ -1977,6 +2116,13 @@ static int vduse_create_dev(struct vduse_dev_config *config,for (u32 i =0 ; i < dev->ngroups; ++i) dev->groups[i].dev = dev; + dev->nas = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->nas; + dev->as =kcalloc (dev->nas,sizeof (dev->as[0 ]),GFP_KERNEL ); +if (!dev->as) + goto err_as; +for (int i =0 ; i < dev->nas; i++) +mutex_init (&dev->as[i].mem_lock); + dev->name =kstrdup (config->name,GFP_KERNEL );if (!dev->name) goto err_str;
@@ -2013,6 +2159,8 @@ static int vduse_create_dev(struct vduse_dev_config *config, err_idr:kfree (dev->name); err_str: +kfree (dev->as); +err_as:kfree (dev->groups); err_vq_groups:vduse_dev_destroy (dev);
@@ -2138,7 +2286,7 @@ static int vduse_dev_init_vdpa(struct vduse_dev *dev, const char *name) vdev = vdpa_alloc_device(struct vduse_vdpa, vdpa, dev->dev, &vduse_vdpa_config_ops, &vduse_map_ops, - dev->ngroups,1 , name, true); + dev->ngroups, dev->nas, name, true);if (IS_ERR (vdev))return PTR_ERR (vdev);
@@ -2153,7 +2301,8 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name, const struct vdpa_dev_set_config *config) {struct vduse_dev *dev; -int ret; +size_t domain_bounce_size; +int ret, i;mutex_lock (&vduse_lock); dev =vduse_find_dev (name);
@@ -2167,29 +2316,38 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,return ret;mutex_lock (&dev->domain_lock); -if (!dev->domain) - dev->domain =vduse_domain_create (VDUSE_IOVA_SIZE -1 , - dev->bounce_size); -mutex_unlock (&dev->domain_lock); -if (!dev->domain) { - ret = -ENOMEM ; - goto domain_err; + ret =0 ; + + domain_bounce_size = dev->bounce_size / dev->nas; +for (i =0 ; i < dev->nas; ++i) { + dev->as[i].domain =vduse_domain_create (VDUSE_IOVA_SIZE -1 , + domain_bounce_size); +if (!dev->as[i].domain) { + ret = -ENOMEM ; + goto err; + } } +mutex_unlock (&dev->domain_lock); + ret =_vdpa_register_device (&dev->vdev->vdpa, dev->vq_num); -if (ret) { - goto register_err; - } +if (ret) + goto err_register;return 0 ; -register_err: +err_register:mutex_lock (&dev->domain_lock); -vduse_domain_destroy (dev->domain); - dev->domain =NULL ; + +err: +for (int j =0 ; j < i; j++) { +if (dev->as[j].domain) { +vduse_domain_destroy (dev->as[j].domain); + dev->as[j].domain =NULL ; + } + }mutex_unlock (&dev->domain_lock); -domain_err:put_device (&dev->vdev->vdpa.dev);return ret;
include/uapi/linux/vduse.h +50 -3
119 lines not on this page · raw message
--
2.51.1
Re: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.
Maybe it's better to document what is protected by RCU and how.
More below.
Quoted 150 lines
Acked-by Jason Wang <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);
I'd expect at least a synchronize_rcu() here to wait for the read is done?
Quoted 85 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);
This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.
Quoted 30 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-2}@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;
drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5
-1if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs);
Same here.
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.
Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>
I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }
Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?
What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?
Quoted 87 lines
Quoted 87 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;
Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?
The function vduse_domain uses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.
Quoted 45 lines
Quoted 30 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-2}@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5
-1if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs);Same here.
Thanks
Re: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?
Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?
See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.
Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?
Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
1) should we need to wait for the DMA to be completed before assigning to the new as
2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue
Quoted 4 lines
The function
vduse_domainuses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.
Thanks
Quoted 47 lines
Quoted 46 lines
Quoted 31 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-2}@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5
-1if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs);Same here.
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()
That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.
I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)
map = token.group->as->domain->bounce_maps[iova];
sz = min_t(size_t, BOUNCE_MAP_SIZE, size);
...
page = token_group->as->domain->bounce_maps
addr = kmap_local_page(page);
do_bounce(map->orig_phys, addr, sz, dir);
kunmap_local(addr);
size -= sz;
iova += sz; }
read_unlock(&token.group->as->domain);
}
Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent. I'm ok if it takes the one before the concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)
map = as->domain->bounce_maps[iova];
sz = min_t(size_t, BOUNCE_MAP_SIZE, size);
...
page = as->domain->bounce_maps
addr = kmap_local_page(page);
do_bounce(map->orig_phys, addr, sz, dir);
kunmap_local(addr);
size -= sz;
iova += sz; }
read_unlock(&as->domain);
}
That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.
Quoted 7 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
Quoted 3 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issue
No, the group will outlive the operation as it is bound to the device.
Quoted 55 lines
Quoted 4 lines
The function
vduse_domainuses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.Thanks
Quoted 47 lines
Quoted 46 lines
Quoted 31 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-2}@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5
-1if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs);Same here.
Thanks
Re: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 341 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;
Quoted 3 lines
}
read_unlock(&token.group->as->domain);
}
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done. But this is not the logic that is implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg(). We need to explain why set_group_asid() doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.
I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;
Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.
I wonder if another group rwlock is sufficient here:
for set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);
for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 6 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueNo, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc. But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
Anyhow, tracking and failing seems to be more robust. For example, kernel swiotlb has a layer to track whether the address that is being unmapped is mapped before. In our case we should make sure when race happens, isolation via ASID won't be broken.
Thanks
Quoted 57 lines
Quoted 56 lines
Quoted 5 lines
The function
vduse_domainuses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.Thanks
Quoted 47 lines
Quoted 46 lines
Quoted 31 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-2}@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5
-1if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs);Same here.
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 348 lines · Jason Wang
Quoted 341 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 3 lines
}
read_unlock(&token.group->as->domain);
}
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.
No, it doesn't need to wait as long as the reader part uses its own copy.
Quoted 3 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().
We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?
Quoted 4 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.
What about:
* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].
Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 40 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
Quoted 8 lines
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.
Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issue
Quoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 4 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID? It seems hard to communicate that the ASID has changed to the DMA operation callback.
Quoted 5 lines
For example,
kernel swiotlb has a layer to track whether the address that is being unmapped is mapped before. In our case we should make sure when race happens, isolation via ASID won't be broken.
Right but it cannot protect if the iotlb is valid in the second ASID but just points to a different physical / virtual page.
Michael S. Tsirkin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 7 lines · Eugenio Perez Martin
Quoted 3 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()?
synchronize_rcu is called after writer makes it's changes.
Quoted 2 lines
What do we need to do on the other side of
syncrhonize_rcu()?
Presumably, return so the caller knows the as has been updated.
However, user-triggerable synchronize_rcu() is almost always a bug.
If that's what is going on, you want srcu.
Signature · 1 line
MST
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Eugenio Perez Martin
Quoted 3 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()?
Quoted 7 lines
synchronize_rcuis called after writer makes it's changes.Quoted 2 lines
What do we need to do on the other side of
syncrhonize_rcu()?Presumably, return so the caller knows the as has been updated.
I'm happy to add the syncrhonize_rcu() just in case, but the caller of vduse_set_group_asid_nomsg does not need to know that the reader has been updated.
The first caller is vduse_dev_reset which has its own way to avoid being called while vqs are being processed. In particular, it reset their addresses which is way more dangerous. I could call it with dev->rwsem down though.
The second one is set_group_asid vdpa callback which is called from the ioctl itself.
Moreover, rcu_assign_pointer is WRITE_ONCE by itself so we know all the readers will get the new value after it. So what's the value of explicitly waiting for all the readers to finalize their DMA operation?
I'd understand if we need to do modifications or memory management in the now unused ASID, but that's not the case here.
Quoted 4 lines
However, user-triggerable
synchronize_rcu()is almost always a bug.If that's what is going on, you want srcu.
I did not know about it, thanks! but I think all the code that can sleep is out of the RCU critical sections now, isn't it?
Michael S. Tsirkin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 23 lines · Eugenio Perez Martin
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Eugenio Perez Martin
Quoted 3 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()?Quoted 7 lines
synchronize_rcuis called after writer makes it's changes.Quoted 2 lines
What do we need to do on the other side of
syncrhonize_rcu()?Presumably, return so the caller knows the as has been updated.
I'm happy to add the
syncrhonize_rcu()just in case, but the caller ofvduse_set_group_asid_nomsgdoes not need to know that the reader has been updated.
Quoted 7 lines
The first caller is
vduse_dev_resetwhich has its own way to avoid being called while vqs are being processed. In particular, it reset their addresses which is way more dangerous. I could call it withdev->rwsemdown though.The second one is
set_group_asidvdpa callback which is called from the ioctl itself.
This one for sure needs to know that after ioctl completed the new AS is in use.
Quoted 2 lines
Moreover,
rcu_assign_pointerisWRITE_ONCEby itself so we know all the readers will get the new value after it.
No we don't. The words "after it" do not have a meaning on an SMP system. The only way to know that another CPU sees a value is to synchronize with it.
Quoted 2 lines
So what's the value of
explicitly waiting for all the readers to finalize their DMA operation?
Read Documentation/memory-barriers.txt
If still unclear, read it again ))
Quoted 10 lines
I'd understand if we need to do modifications or memory management in the now unused ASID, but that's not the case here.
Quoted 4 lines
However, user-triggerable
synchronize_rcu()is almost always a bug.If that's what is going on, you want srcu.
I did not know about it, thanks! but I think all the code that can sleep is out of the RCU critical sections now, isn't it?
what I tried to say is that if triggering synchronize_rcu from userspace is a problem, srcu is one solution.
Re: [PATCH v9 5/6] vduse: add vq group asid support
In reply to Eugenio Perez MartinQuoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}
Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.
Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?
Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().
Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.
Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.
See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);
Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 34 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
Quoted 4 lines
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Basically what I meant, assuming group0.as = as0
cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)
cpu0 may read as1 while it wants as0 actually?
Quoted 13 lines
Quoted 5 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 5 lines
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?
If possible, it would be better.
Quoted 2 lines
It seems hard to
communicate that the ASID has changed to the DMA operation callback.
Maybe we can encode this into iova.
Quoted 10 lines
Quoted 6 lines
For example,
kernel swiotlb has a layer to track whether the address that is being unmapped is mapped before. In our case we should make sure when race happens, isolation via ASID won't be broken.Right but it cannot protect if the iotlb is valid in the second ASID but just points to a different physical / virtual page.
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once
set_group_asid()return, there should still be a pending DMA that is using the old as.Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?Usually we don't need special care on the read side. But as discussed,
synchronize_rcu()is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to usesmp_store_release()andsmp_load_acqurie().Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 34 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
Quoted 4 lines
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 8 lines
Basically what I meant, assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)cpu0 may read as1 while it wants as0 actually?
Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.
There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.
From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.
Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:
void* thread0_func(void* arg) {
struct {
int vq_group = 0,
int iova, size, perm, ...
} s;
int fd = (intptr_t)arg;
ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
// TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
// TODO: Wait until thread0 complete SET_GROUP_ASID
ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);
return NULL;
}void* thread1_func(void* arg) {
struct vhost_vring_state s = {
.index = 0,
.num = 1,
};
int fd = (int)(intptr_t)arg;
// TODO: Wait until thread2 calls dma_map
ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed with dma_unmap
return NULL;
}int main() {
pthread_t thread0, thread1;
int fd = open("/dev/vhost-vdpa-0", ...)
pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd);
pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd);
pthread_join(thread1, NULL);
pthread_join(thread2, NULL);
return EXIT_SUCCESS;
}---
We need something to synchronize at userland level here, filling the TODOs.
We can replace the hypothetical VHOST_VDPA_VQ_GROUP_DMA_MAP and _UNMAP with access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.
We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So DMA_MAP and DMA_UNMAP does not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?
If possible, it would be better.
It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.
Quoted 5 lines
Quoted 2 lines
It seems hard to
communicate that the ASID has changed to the DMA operation callback.Maybe we can encode this into iova.
I'm adding the full rwlock, but can you expand on your idea on this? It would be great to have it documented in case we need future optimizations.
Re: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 528 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once
set_group_asid()return, there should still be a pending DMA that is using the old as.Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?Usually we don't need special care on the read side. But as discussed,
synchronize_rcu()is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to usesmp_store_release()andsmp_load_acqurie().Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 34 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
Quoted 4 lines
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 8 lines
Basically what I meant, assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)cpu0 may read as1 while it wants as0 actually?
Quoted 8 lines
Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.
There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.
Probably.
Quoted 27 lines
From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to
set_group_asiddoes not need to access the vq group.Now let's say that we add that
vdpa_opscallback (and ioctls) that maps and unmap based on avq_group. And all of the operations (dma_map,set_group_asid, anddma_unmap) are serialized by taking the same mutex. cpu0 still maydma_unmapover as0 ifset_group_asidis not properly serialized at vhost/vdpa or userland level:void*
thread0_func(void* arg){struct { int vq_group = 0, int iova, size, perm, ... } s; int fd = (intptr_t)arg; ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s); // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID // TODO: Wait until thread0 complete SET_GROUP_ASID ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data); return NULL; }
Quoted 8 lines
void* thread1_func(void* arg) {
struct vhost_vring_state s = {
.index = 0,
.num = 1,
};
int fd = (int)(intptr_t)arg;
// TODO: Wait until thread2 calls dma_mapThis is something exactly rwlock or synchronize_rcu() can do? Or is this the charge of the vDPA parent to do that?
If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via mmap() or umem_reg.
Looking at existing implementation:
- for mlx5e, it looks like it assumes the set_group_asid() works only without DRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated
Quoted 2 lines
ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed withdma_unmap
But the issue I mention is that the, from the view of the vDPA bus:
1) it offers set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is used
So theoretically, set_group_as_id() could happen between
dma_addr = dma_map();
and
dma_unmap(dma_adrr);
But those two dma_addr refers to the different address space. Instead of trying to do synchronization, maybe we can simply fail set_group_asid if DRIVER_OK is set.
Quoted 26 lines
return NULL; }int main() { pthread_t thread0, thread1; int fd = open("/dev/vhost-vdpa-0", ...) pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd); pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd); pthread_join(thread1, NULL); pthread_join(thread2, NULL); return EXIT_SUCCESS; }---
We need something to synchronize at userland level here, filling the TODOs.
We can replace the hypothetical
VHOST_VDPA_VQ_GROUP_DMA_MAPand_UNMAPwith access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.
Quoted 5 lines
We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So
DMA_MAPandDMA_UNMAPdoes not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.
See above.
Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?
If possible, it would be better.
It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
It seems hard to
communicate that the ASID has changed to the DMA operation callback.Maybe we can encode this into iova.
I'm adding the full rwlock, but can you expand on your idea on this?
Encode the asid to the upper bits of IOVA, so when doing dma_unmap() we can compare the group/as with the one that is encoded in dma_addr. If it differs, warn or bug.
If we can make sure there's no set_group_asid() when DRIVER_OK is set, it's not a must then (or could be treated as a kind of hardening).
Quoted 3 lines
It would be great to have it documented in case we need future optimizations.
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 575 lines · Jason Wang
Quoted 528 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once
set_group_asid()return, there should still be a pending DMA that is using the old as.Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?Usually we don't need special care on the read side. But as discussed,
synchronize_rcu()is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to usesmp_store_release()andsmp_load_acqurie().Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 34 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
Quoted 4 lines
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 8 lines
Basically what I meant, assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)cpu0 may read as1 while it wants as0 actually?
Quoted 8 lines
Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.
There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.
Probably.
Quoted 34 lines
From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to
set_group_asiddoes not need to access the vq group.Now let's say that we add that
vdpa_opscallback (and ioctls) that maps and unmap based on avq_group. And all of the operations (dma_map,set_group_asid, anddma_unmap) are serialized by taking the same mutex. cpu0 still maydma_unmapover as0 ifset_group_asidis not properly serialized at vhost/vdpa or userland level:void*
thread0_func(void* arg){struct { int vq_group = 0, int iova, size, perm, ... } s; int fd = (intptr_t)arg; ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s); // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID // TODO: Wait until thread0 complete SET_GROUP_ASID ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data); return NULL; }void* thread1_func(void* arg) { struct vhost_vring_state s = { .index = 0, .num = 1, }; int fd = (int)(intptr_t)arg;
Quoted 5 lines
Quoted 1 line
// TODO: Wait until thread2 calls dma_mapThis is something exactly rwlock or
synchronize_rcu()can do? Or is this the charge of the vDPA parent to do that?
No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.
If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:
cpu0] DMA map ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
Now we have two possibilities: either cpu0] DMA_UNMAP is called or cpu1] set_group_asid is called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:
cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock
If set_group_asid run first by chance, ASID 1 is the one that is updated:
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock
cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the set_map call. I'm not adding the synchronize_rcu call because we stated it should not be called from userland, but the outcome is similar.
Let me put another example: It's like calling fdup() and write() from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of the writes() is just not predictable. And the kernel just cannot make it predictable.
static int fd_a;
static int fd_b;
void *write_thread(void *arg) {
const char *msg = "Writing to fd_b\n";
write(fd_b, msg, strlen(msg));
return NULL;
}void *dup_thread(void *arg) {
dup2(fd_a, fd_b);
return NULL;
}int main() {
pthread_t writer, duper;
fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644);
fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644);
pthread_create(&writer, NULL, write_thread, NULL);
pthread_create(&duper, NULL, dup_thread, NULL);
pthread_join(writer, NULL);
pthread_join(duper, NULL);
close(fd_a);
close(fd_b);
return 0;
}--
Quoted 11 lines
If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via
mmap()orumem_reg.Looking at existing implementation:
- for mlx5e, it looks like it assumes the
set_group_asid()works only withoutDRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated
Yes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.
Quoted 10 lines
Quoted 2 lines
ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed withdma_unmapBut the issue I mention is that the, from the view of the vDPA bus:
1) it offers
set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is usedSo theoretically,
set_group_as_id()could happen between
Quoted 7 lines
dma_addr=dma_map();and
dma_unmap(dma_adrr);But those two
dma_addrrefers to the different address space.
I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?
Quoted 4 lines
Instead
of trying to do synchronization, maybe we can simply failset_group_asidifDRIVER_OKis set.
That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.
Quoted 64 lines
Quoted 26 lines
return NULL; }int main() { pthread_t thread0, thread1; int fd = open("/dev/vhost-vdpa-0", ...) pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd); pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd); pthread_join(thread1, NULL); pthread_join(thread2, NULL); return EXIT_SUCCESS; }---
We need something to synchronize at userland level here, filling the TODOs.
We can replace the hypothetical
VHOST_VDPA_VQ_GROUP_DMA_MAPand_UNMAPwith access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.Quoted 5 lines
We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So
DMA_MAPandDMA_UNMAPdoes not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.See above.
Quoted 30 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?
If possible, it would be better.
It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.
Quoted 3 lines
Quoted 2 lines
It seems hard to
communicate that the ASID has changed to the DMA operation callback.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Maybe we can encode this into iova.
I'm adding the full rwlock, but can you expand on your idea on this?
Encode the asid to the upper bits of IOVA, so when doing
dma_unmap()we can compare the group/as with the one that is encoded indma_addr. If it differs, warn or bug.
But the ASID is already a parameter in the dma_unmap. I thought you meant to encode in the device's memory read.
Quoted 9 lines
If we can make sure there's no
set_group_asid()whenDRIVER_OKis set, it's not a must then (or could be treated as a kind of hardening).Quoted 3 lines
It would be great to have it documented in case we need future optimizations.
Thanks
Re: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 554 lines · Eugenio Perez Martin
Quoted 552 lines · Jason Wang
Quoted 528 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once
set_group_asid()return, there should still be a pending DMA that is using the old as.Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?Usually we don't need special care on the read side. But as discussed,
synchronize_rcu()is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to usesmp_store_release()andsmp_load_acqurie().Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 34 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
Quoted 4 lines
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 8 lines
Basically what I meant, assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)cpu0 may read as1 while it wants as0 actually?
Quoted 8 lines
Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.
There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.
Probably.
Quoted 11 lines
From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to
set_group_asiddoes not need to access the vq group.Now let's say that we add that
vdpa_opscallback (and ioctls) that maps and unmap based on avq_group. And all of the operations (dma_map,set_group_asid, anddma_unmap) are serialized by taking the same mutex. cpu0 still maydma_unmapover as0 ifset_group_asidis not properly serialized at vhost/vdpa or userland level:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
void*
thread0_func(void* arg){struct { int vq_group = 0, int iova, size, perm, ... } s; int fd = (intptr_t)arg; ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
I don't get the semantic of VHOST_VDPA_VQ_GROUP_DMA_MAP. And it's probably wrong to allow userspace to map based on group.
We have VHOST_IOTLB_UPDATE which map pages based on as (asid) this is correct.
More below
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
// TODO: Signal thread0 that it can proceed with SET_GROUP_ASID // TODO: Wait until thread0 complete SET_GROUP_ASID ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data); return NULL; }void* thread1_func(void* arg) { struct vhost_vring_state s = { .index = 0, .num = 1, }; int fd = (int)(intptr_t)arg; // TODO: Wait until thread2 calls dma_map
Quoted 7 lines
Quoted 3 lines
This is something exactly rwlock or
synchronize_rcu()can do? Or is this the charge of the vDPA parent to do that?No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.
Ok, basically I think we are talking about different things. That's fine.
If I understand you correctly, you mean we need to synchronize between IOTLB updating (IOTLB_UPDATE/IOTLB_INVALIDATE) and set_group_asid()?
This sounds unnecessary since:
1) IOTLB_UPDATE/IOTLB_INVALIDATE is updating the address space internal mappings
2) set_group_asid() is to assign an AS to a group
Quoted 35 lines
If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:
cpu0] DMA map ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lockNow we have two possibilities: either cpu0]
DMA_UNMAPis called or cpu1]set_group_asidis called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:cpu0] DMA unmap ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lock
cpu1]set_group_asidioctl call in userland
-> Takewrite_lockin the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlockwrite_lockIf
set_group_asidrun first by chance, ASID 1 is the one that is updated:
cpu1]set_group_asidioctl call in userland
-> Takewrite_lockin the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlockwrite_lock
cpu0] DMA unmap ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lock
Quoted 7 lines
On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the
set_mapcall. I'm not adding thesynchronize_rcucall because we stated it should not be called from userland, but the outcome is similar.
I think we should figure out if VHOST_VDPA_VQ_GROUP_DMA_MAP is useful or not.
Quoted 72 lines
Let me put another example: It's like calling
fdup()andwrite()from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of thewrites()is just not predictable. And the kernel just cannot make it predictable.static int
fd_a;
static intfd_b;void *write_thread(void *arg) { const char *msg = "Writing to fd_b\n"; write(fd_b, msg, strlen(msg)); return NULL; }void *dup_thread(void *arg) { dup2(fd_a, fd_b); return NULL; }int main() { pthread_t writer, duper; fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644); fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644); pthread_create(&writer, NULL, write_thread, NULL); pthread_create(&duper, NULL, dup_thread, NULL); pthread_join(writer, NULL); pthread_join(duper, NULL); close(fd_a); close(fd_b); return 0; }--
Quoted 11 lines
If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via
mmap()orumem_reg.Looking at existing implementation:
- for mlx5e, it looks like it assumes the
set_group_asid()works only withoutDRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulatedYes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.
Quoted 16 lines
Quoted 2 lines
ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed withdma_unmapBut the issue I mention is that the, from the view of the vDPA bus:
1) it offers
set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is usedSo theoretically,
set_group_as_id()could happen between
dma_addr=dma_map();and
dma_unmap(dma_adrr);
Quoted 7 lines
Quoted 1 line
But those two
dma_addrrefers to the different address space.I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?
Just to clarify, I for dma_map()/dma_unmap() this is not the UAPI part (and we don't have that). I basically mean the dma API which is used by virtio-vDPA.
Quoted 10 lines
Quoted 5 lines
Instead
of trying to do synchronization, maybe we can simply failset_group_asidifDRIVER_OKis set.That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.
Did you mean the SVQ may update the group asid while DRIVER_OK is set? If yes, we need to fix that.
Quoted 70 lines
Quoted 70 lines
Quoted 27 lines
return NULL; }int main() { pthread_t thread0, thread1; int fd = open("/dev/vhost-vdpa-0", ...) pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd); pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd); pthread_join(thread1, NULL); pthread_join(thread2, NULL); return EXIT_SUCCESS; }---
We need something to synchronize at userland level here, filling the TODOs.
We can replace the hypothetical
VHOST_VDPA_VQ_GROUP_DMA_MAPand_UNMAPwith access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.Quoted 5 lines
We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So
DMA_MAPandDMA_UNMAPdoes not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.See above.
Quoted 34 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?
If possible, it would be better.
It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.
Quoted 5 lines
Quoted 2 lines
It seems hard to
communicate that the ASID has changed to the DMA operation callback.Maybe we can encode this into iova.
I'm adding the full rwlock, but can you expand on your idea on this?
Quoted 7 lines
Quoted 4 lines
Encode the asid to the upper bits of IOVA, so when doing
dma_unmap()we can compare the group/as with the one that is encoded indma_addr. If it differs, warn or bug.But the ASID is already a parameter in the
dma_unmap. I thought you meant to encode in the device's memory read.
To avoid unmap a address that is not belong to the this ASID.
Quoted 10 lines
Quoted 10 lines
If we can make sure there's no
set_group_asid()whenDRIVER_OKis set, it's not a must then (or could be treated as a kind of hardening).Quoted 3 lines
It would be great to have it documented in case we need future optimizations.
Thanks
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 525 lines · Jason Wang
Quoted 523 lines · Eugenio Perez Martin
Quoted 521 lines · Jason Wang
Quoted 519 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by Jason Wang <[email protected]>I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +20
-6@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; }Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39
-6+} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ;Quoted 13 lines
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922
Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4
-2vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once
set_group_asid()return, there should still be a pending DMA that is using the old as.Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?Usually we don't need special care on the read side. But as discussed,
synchronize_rcu()is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to usesmp_store_release()andsmp_load_acqurie().Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 38 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Basically what I meant, assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)
[1]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
cpu0 may read as1 while it wants as0 actually?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.
There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.
[2]
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Probably.
Quoted 5 lines
From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to
set_group_asiddoes not need to access the vq group.
[3]
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Now let's say that we add that
vdpa_opscallback (and ioctls) that maps and unmap based on avq_group. And all of the operations (dma_map,set_group_asid, anddma_unmap) are serialized by taking the same mutex. cpu0 still maydma_unmapover as0 ifset_group_asidis not properly serialized at vhost/vdpa or userland level:void*
thread0_func(void* arg){
struct {
intvq_group= 0,
int iova, size, perm, ...
} s;
int fd = (intptr_t)arg;
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);I don't get the semantic of
VHOST_VDPA_VQ_GROUP_DMA_MAP. And it's probably wrong to allow userspace to map based on group.We have
VHOST_IOTLB_UPDATEwhich map pages based on as (asid) this is correct.
Yes, I said that at [2], in reply to your scenario at [1] :). Did I misunderstand?
Quoted 29 lines
More below
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
// TODO: Signal thread0 that it can proceed with SET_GROUP_ASID // TODO: Wait until thread0 complete SET_GROUP_ASID ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data); return NULL; }void* thread1_func(void* arg) { struct vhost_vring_state s = { .index = 0, .num = 1, }; int fd = (int)(intptr_t)arg; // TODO: Wait until thread2 calls dma_mapQuoted 7 lines
Quoted 3 lines
This is something exactly rwlock or
synchronize_rcu()can do? Or is this the charge of the vDPA parent to do that?No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.
Ok, basically I think we are talking about different things. That's fine.
Quoted 8 lines
If I understand you correctly, you mean we need to synchronize between IOTLB updating (IOTLB_UPDATE/IOTLB_INVALIDATE) and
set_group_asid()?This sounds unnecessary since:
1) IOTLB_UPDATE/IOTLB_INVALIDATE is updating the address space internal mappings
2)set_group_asid()is to assign an AS to a group
Right, we're on the same page here [3].
Quoted 43 lines
Quoted 35 lines
If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:
cpu0] DMA map ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lockNow we have two possibilities: either cpu0]
DMA_UNMAPis called or cpu1]set_group_asidis called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:cpu0] DMA unmap ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lock
cpu1]set_group_asidioctl call in userland
-> Takewrite_lockin the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlockwrite_lockIf
set_group_asidrun first by chance, ASID 1 is the one that is updated:
cpu1]set_group_asidioctl call in userland
-> Takewrite_lockin the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlockwrite_lock
cpu0] DMA unmap ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lockQuoted 7 lines
On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the
set_mapcall. I'm not adding thesynchronize_rcucall because we stated it should not be called from userland, but the outcome is similar.
Quoted 2 lines
I think we should figure out if
VHOST_VDPA_VQ_GROUP_DMA_MAPis useful or not.
It's not. I just wanted to expand how I understood your scenario, but let me know if I missed something!
Quoted 80 lines
Quoted 72 lines
Let me put another example: It's like calling
fdup()andwrite()from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of thewrites()is just not predictable. And the kernel just cannot make it predictable.static int
fd_a;
static intfd_b;void *write_thread(void *arg) { const char *msg = "Writing to fd_b\n"; write(fd_b, msg, strlen(msg)); return NULL; }void *dup_thread(void *arg) { dup2(fd_a, fd_b); return NULL; }int main() { pthread_t writer, duper; fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644); fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644); pthread_create(&writer, NULL, write_thread, NULL); pthread_create(&duper, NULL, dup_thread, NULL); pthread_join(writer, NULL); pthread_join(duper, NULL); close(fd_a); close(fd_b); return 0; }--
Quoted 11 lines
If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via
mmap()orumem_reg.Looking at existing implementation:
- for mlx5e, it looks like it assumes the
set_group_asid()works only withoutDRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulatedYes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.
Quoted 16 lines
Quoted 2 lines
ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed withdma_unmapBut the issue I mention is that the, from the view of the vDPA bus:
1) it offers
set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is usedSo theoretically,
set_group_as_id()could happen between
dma_addr=dma_map();and
dma_unmap(dma_adrr);Quoted 7 lines
Quoted 1 line
But those two
dma_addrrefers to the different address space.I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?
Quoted 4 lines
Just to clarify, I for dma_map()/dma_unmap() this is not the UAPI part (and we don't have that). I basically mean the dma API which is used by virtio-vDPA.
So let's say that the virtio_vdpa driver is able to call .set_group_asid just to explain how much it is protected:
->vduse_dev_map_page and ->vduse_dev_unmap_page
In this series, rcu_dereference makes sure that only the stack copy of the domain pointer is used, but the next version the code will have a read_lock taken protecting that set_group_asid actions on token.group->as cannot run at the same time that the vduse_domain_map_page(...) or vduse_domain_unmap_page(...) function call run in cpu0:
static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs)
{
struct vduse_iova_domain *domain;
dma_addr_t r;
if (!token.group)
return DMA_MAPPING_ERROR;
read_lock(&dev->groups[group].as_lock);
domain = token.group->as->domain;
r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);
read_unlock(&dev->groups[group].as_lock);
return r;
}static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
size_t size, enum dma_data_direction dir, unsigned long attrs)
{
struct vduse_iova_domain *domain;
if (!token.group)
return;
read_lock(&dev->groups[group].as_lock);
domain = token.group->as->domain;
vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
read_unlock(&dev->groups[group].as_lock);
}---
-> vduse_set_group_asid will have the corresponding write_lock, so it cannot modify any vq group AS pointer while they're mapping or unmapping pages from a domain:
static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
unsigned int group, unsigned int asid)
{
write_lock(&dev->groups[group].as_lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].as_lock);
}---
Now your question:
assuming group0.as = as0
cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)
cpu0 may read as1 while it wants as0 actually?
--
Yes, it can happen even with the rwlock or any other synchronization in VDUSE, as it does not order the vdpa core calls.
This is the code I have in mind to prove this, let me know if I'm missing something from your scenario:
static int thread1_func(void *data)
{
vduse_dev_map_page(token, ...);
printk(KERN_INFO "map_page called\n");
// TODO: Need a way to know if the group ASID has changed to
// know if this unmap is valid or not.
vduse_dev_unmap_page(token, ...);
printk(KERN_INFO "unmap_page called\n");
return 0;
}static int thread2_func(void *data)
{
u32 vq_group = 0;
u32 asid = 1;
vdpa->ops->set_group_asid(global_vdpa, vq_group, asid);
printk(KERN_INFO "set_group_asid called\n");
return 0;
}static int __init kernel_threads_init(void)
{
struct task_struct *thread1, *thread2;
thread1 = kthread_run(thread1_func, NULL, "vdpa_map_unmap");
thread2 = kthread_run(thread2_func, NULL, "vdpa_set_asid");
return 0;
}---
I keep thinking that this is a race condition in the hypothetical vdpa core or the module that runs this code, not solvable in VDUSE. We need to add barriers at this level, not VDUSE.
But I think it is a good idea to add some code in VDUSE to mitigate it. Maybe just adding the expected vq queue index to vdev->map->map_page and vdev->map->unmap_page? This way we can issue an error if the race happens.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Instead
of trying to do synchronization, maybe we can simply failset_group_asidifDRIVER_OKis set.
Quoted 8 lines
Quoted 4 lines
That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.
Did you mean the SVQ may update the group asid while
DRIVER_OKis set? If yes, we need to fix that.
Not at this moment.
Quoted 93 lines
Quoted 70 lines
Quoted 70 lines
Quoted 27 lines
return NULL; }int main() { pthread_t thread0, thread1; int fd = open("/dev/vhost-vdpa-0", ...) pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd); pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd); pthread_join(thread1, NULL); pthread_join(thread2, NULL); return EXIT_SUCCESS; }---
We need something to synchronize at userland level here, filling the TODOs.
We can replace the hypothetical
VHOST_VDPA_VQ_GROUP_DMA_MAPand_UNMAPwith access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.Quoted 5 lines
We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So
DMA_MAPandDMA_UNMAPdoes not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.See above.
Quoted 34 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.
No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Anyhow, tracking and failing seems to be more robust.
I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?
If possible, it would be better.
It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.
Quoted 5 lines
Quoted 2 lines
It seems hard to
communicate that the ASID has changed to the DMA operation callback.Maybe we can encode this into iova.
I'm adding the full rwlock, but can you expand on your idea on this?
Quoted 7 lines
Quoted 4 lines
Encode the asid to the upper bits of IOVA, so when doing
dma_unmap()we can compare the group/as with the one that is encoded indma_addr. If it differs, warn or bug.But the ASID is already a parameter in the
dma_unmap. I thought you meant to encode in the device's memory read.To avoid unmap a address that is not belong to the this ASID.
Quoted 10 lines
Quoted 10 lines
If we can make sure there's no
set_group_asid()whenDRIVER_OKis set, it's not a must then (or could be treated as a kind of hardening).Quoted 3 lines
It would be great to have it documented in case we need future optimizations.
Thanks
Thanks
Re: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 576 lines · Eugenio Perez Martin
Quoted 525 lines · Jason Wang
Quoted 523 lines · Eugenio Perez Martin
Quoted 521 lines · Jason Wang
Quoted 519 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
Add support for assigning Address Space Identifiers (ASIDs) to each VQ
group. This enables mapping each group into a distinct memory space.Now that the driver can change ASID in the middle of operation, the
domain that each vq address point is also protected by domain_lock.Maybe it's better to document what is protected by RCU and how.
I added the
_rcuannotation but I can expand it for sure. I can also modify the commit message.Quoted 4 lines
More below.
Quoted 2 lines
Acked-by: Jason Wang <[email protected]>
I forgot to remove this, my bad!
Quoted 148 lines
Quoted 148 lines
Signed-off-by: Eugenio Pérez <[email protected]>
---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic
context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to
justify it.v7:
* Take write lock in the error path (Jason).v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/
(MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD
ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in
vduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z)
return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).v3:
* Get the vduse domain through the vduse_as in the map functions
(Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number
(Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower
value to reduce memory consumption, but vqs are already limited to
that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with
VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between
ASIDs.RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first
part of the struct is the same.
---
drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
include/uapi/linux/vduse.h | 53 ++++-
2 files changed, 314 insertions(+), 109 deletions(-)diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 97be04f73fbf..ff95ed56f22d 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -11,6 +11,7 @@
#include "linux/virtio_net.h"
#include <linux/init.h>
#include <linux/module.h>
+#include <linux/rcupdate.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/eventfd.h>
@@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)
#define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
struct mm_struct *mm;
};+struct vduse_as {
+ struct vduse_iova_domain *domain;
+ struct vduse_umem *umem;
+ struct mutex mem_lock;
+};
+
struct vduse_vq_group {
+ struct vduse_as *as __rcu;
struct vduse_dev *dev;
};@@ -94,7 +103,7 @@ struct vduse_dev {
struct vduse_vdpa *vdev;
struct device *dev;
struct vduse_virtqueue **vqs;
- struct vduse_iova_domain *domain;
+ struct vduse_as *as;
char *name;
struct mutex lock;
spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
u32 vq_num;
u32 vq_align;
u32 ngroups;
- struct vduse_umem *umem;
+ u32 nas;
struct vduse_vq_group *groups;
- struct mutex mem_lock;
unsigned int bounce_size;
struct mutex domain_lock;
};
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
return vduse_dev_msg_sync(dev, &msg);
}-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
u64 start, u64 last)
{
struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
return -EINVAL;msg.req.type = VDUSE_UPDATE_IOTLB;
- msg.req.iova.start = start;
- msg.req.iova.last = last;
+ if (dev->api_version < VDUSE_API_VERSION_1) {
+ msg.req.iova.start = start;
+ msg.req.iova.last = last;
+ } else {
+ msg.req.iova_v2.start = start;
+ msg.req.iova_v2.last = last;
+ msg.req.iova_v2.asid = asid;
+ }return vduse_dev_msg_sync(dev, &msg);
}
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
return mask;
}Quoted 13 lines
Quoted 10 lines
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+ unsigned int group, unsigned int asid)
+{
+ /*
+ * Two concurrent updates to this pointer are valid as they cannot
+ * point to an invalid region. It is ok for them to race as long as
+ * the readers see a consistent state through RCU.
+ */
+ rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);I'd expect at least a
synchronize_rcu()here to wait for the read is done?Quoted 3 lines
What's the use? The only thing left here is to return from
vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?See below.
Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
+}
+
static void vduse_dev_reset(struct vduse_dev *dev)
{
int i;
- struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */
- if (domain && domain->bounce_map)
- vduse_domain_reset_bounce_map(domain);
+ for (i = 0; i < dev->nas; i++) {
+ struct vduse_iova_domain *domain = dev->as[i].domain;
+
+ if (domain && domain->bounce_map)
+ vduse_domain_reset_bounce_map(domain);
+ }
+
+ for (i = 0; i < dev->ngroups; i++)
+ vduse_set_group_asid_nomsg(dev, i, 0);down_write(&dev->rwsem);
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
return ret;
}+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+ unsigned int asid)
+{
+ struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+ struct vduse_dev_msg msg = { 0 };
+ int r;
+
+ if (dev->api_version < VDUSE_API_VERSION_1 ||
+ group >= dev->ngroups || asid >= dev->nas)
+ return -EINVAL;
+
+ msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+ msg.req.vq_group_asid.group = group;
+ msg.req.vq_group_asid.asid = asid;
+
+ r = vduse_dev_msg_sync(dev, &msg);
+ if (r < 0)
+ return r;
+
+ vduse_set_group_asid_nomsg(dev, group, asid);
+ return 0;
+}
+
static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
struct vdpa_vq_state *state)
{
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
struct vduse_dev *dev = vdpa_to_vduse(vdpa);
int ret;- ret = vduse_domain_set_map(dev->domain, iotlb);
+ ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
if (ret)
return ret;- ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+ ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
if (ret) {
- vduse_domain_clear_map(dev->domain, iotlb);
+ vduse_domain_clear_map(dev->as[asid].domain, iotlb);
return ret;
}@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
.get_vq_affinity = vduse_vdpa_get_vq_affinity,
.reset = vduse_vdpa_reset,
.set_map = vduse_vdpa_set_map,
+ .set_group_asid = vduse_set_group_asid,
.get_vq_map = vduse_get_vq_map,
.free = vduse_vdpa_free,
};
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
enum dma_data_direction dir)
{
struct vduse_dev *vdev;
+ struct vduse_as *as;
struct vduse_iova_domain *domain;if (!token.group)
return;Quoted 13 lines
Quoted 8 lines
vdev = token.group->dev;
- domain = vdev->domain;
-
+ rcu_read_lock();
+ as = rcu_dereference(token.group->as);
+ domain = as->domain;
+ rcu_read_unlock();
vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);This is suspicious, at least we should do
rcu_read_unlock()aftervduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.Quoted 7 lines
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.
Quoted 9 lines
But I meant what happens if
[cpu0]rcu_read_lock()
[cpu0]as =rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()That should go ok. What I'm trying to protect here is the iterations in
vduse_domain_sync_single_for_device->vduse_domain_bounce.I'm going to embed that function here in
vduse_dev_sync_single_for_deviceand omit RCU and some details to make the point easier:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
read_lock(&token.group->as->domain);
while (size)map = token.group->as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = token_group->as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;}
read_unlock(&token.group->as->domain);
}Quoted 6 lines
Quoted 4 lines
Right, so I meant for rwlock like semantic (let's forget the sleeping here).
vduse_set_group_asid_nomsg()should use "write lock" so it must wait for the "read lock" to be done.No, it doesn't need to wait as long as the reader part uses its own copy.
It probably won't crash but I meant if we have logic issues. For example, once
set_group_asid()return, there should still be a pending DMA that is using the old as.Quoted 9 lines
Quoted 4 lines
But this is not the logic that is
implemented in this patch as there's nosynchronize_rcu()in thevduse_set_group_asid_nomsg().We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after
syncrhonize_rcu()? What do we need to do on the other side ofsyncrhonize_rcu()?Usually we don't need special care on the read side. But as discussed,
synchronize_rcu()is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to usesmp_store_release()andsmp_load_acqurie().Quoted 6 lines
Quoted 5 lines
We need to explain why
set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.Quoted 7 lines
What about:
* It does not matter if other thread modify
group->asas long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modifygroup->asas long as the update is atomic.See above reply.
Quoted 71 lines
?
Quoted 10 lines
Quoted 10 lines
Now, depending on the point where another execution thread changes
token_group->asand how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].Quoted 9 lines
Quoted 4 lines
And I guess there are other possible outcomes too.
So I need to make sure that the pointer accesses in all
vduse_domain_bounceis coherent.I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?
The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.
Quoted 45 lines
Quoted 21 lines
I'm ok if it takes the one before the
concurrent call tovduse_set_group_asid_nomsgor the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:
vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...){
as =token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this.read_lock(&as->domain);
while (size)map = as->domain->bounce_maps[iova]; sz = min_t(size_t, BOUNCE_MAP_SIZE, size); ... page = as->domain->bounce_maps addr = kmap_local_page(page); do_bounce(map->orig_phys, addr, sz, dir); kunmap_local(addr); size -= sz; iova += sz;Quoted 8 lines
}
read_unlock(&as->domain);
}That can be done in many ways. Probably the
read_lockis already enough but it is not explicit that it is protectingtoken.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.I wonder if another group rwlock is sufficient here:
for
set_group_as_id()
write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];write_unlock(&dev->groups[group].lock);for the case where we need defer as
read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using asread_unlock(&dev->groups[group].lock);Quoted 7 lines
Quoted 3 lines
If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?
A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.
Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.
Quoted 38 lines
Another disadvantage of the lock vs RCU or
READ_ONCEis that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.
It's basically a question that
Quoted 7 lines
Quoted 2 lines
1) should we need to wait for the DMA to be completed before assigning to the new as
I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.
See below.
Quoted 5 lines
Quoted 4 lines
2) should we track the
set_group_asid()for the group that has pending DMA to avoid potential issueQuoted 6 lines
Quoted 1 line
No, the group will outlive the operation as it is bound to the device.
I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.
I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Basically what I meant, assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)[1]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
cpu0 may read as1 while it wants as0 actually?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.
There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.
[2]
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Probably.
Quoted 5 lines
From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to
set_group_asiddoes not need to access the vq group.[3]
Quoted 15 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Now let's say that we add that
vdpa_opscallback (and ioctls) that maps and unmap based on avq_group. And all of the operations (dma_map,set_group_asid, anddma_unmap) are serialized by taking the same mutex. cpu0 still maydma_unmapover as0 ifset_group_asidis not properly serialized at vhost/vdpa or userland level:void*
thread0_func(void* arg){struct { int vq_group = 0, int iova, size, perm, ... } s; int fd = (intptr_t)arg; ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
Quoted 8 lines
Quoted 5 lines
I don't get the semantic of
VHOST_VDPA_VQ_GROUP_DMA_MAP. And it's probably wrong to allow userspace to map based on group.We have
VHOST_IOTLB_UPDATEwhich map pages based on as (asid) this is correct.Yes, I said that at [2], in reply to your scenario at [1] :). Did I misunderstand?
Nope, it looks like it's me that misread your reply.
Quoted 286 lines
Quoted 30 lines
More below
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
// TODO: Signal thread0 that it can proceed with SET_GROUP_ASID // TODO: Wait until thread0 complete SET_GROUP_ASID ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data); return NULL; }void* thread1_func(void* arg) { struct vhost_vring_state s = { .index = 0, .num = 1, }; int fd = (int)(intptr_t)arg; // TODO: Wait until thread2 calls dma_mapQuoted 7 lines
Quoted 3 lines
This is something exactly rwlock or
synchronize_rcu()can do? Or is this the charge of the vDPA parent to do that?No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.
Ok, basically I think we are talking about different things. That's fine.
Quoted 8 lines
If I understand you correctly, you mean we need to synchronize between IOTLB updating (IOTLB_UPDATE/IOTLB_INVALIDATE) and
set_group_asid()?This sounds unnecessary since:
1) IOTLB_UPDATE/IOTLB_INVALIDATE is updating the address space internal mappings
2)set_group_asid()is to assign an AS to a groupRight, we're on the same page here [3].
Quoted 43 lines
Quoted 35 lines
If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:
cpu0] DMA map ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lockNow we have two possibilities: either cpu0]
DMA_UNMAPis called or cpu1]set_group_asidis called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:cpu0] DMA unmap ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lock
cpu1]set_group_asidioctl call in userland
-> Takewrite_lockin the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlockwrite_lockIf
set_group_asidrun first by chance, ASID 1 is the one that is updated:
cpu1]set_group_asidioctl call in userland
-> Takewrite_lockin the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlockwrite_lock
cpu0] DMA unmap ioctl call in userland
-> Takeread_lockin the vduse module
-> Update the IOTLB tree of ASID 0
-> unlockread_lockQuoted 7 lines
On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the
set_mapcall. I'm not adding thesynchronize_rcucall because we stated it should not be called from userland, but the outcome is similar.Quoted 2 lines
I think we should figure out if
VHOST_VDPA_VQ_GROUP_DMA_MAPis useful or not.It's not. I just wanted to expand how I understood your scenario, but let me know if I missed something!
Quoted 80 lines
Quoted 72 lines
Let me put another example: It's like calling
fdup()andwrite()from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of thewrites()is just not predictable. And the kernel just cannot make it predictable.static int
fd_a;
static intfd_b;void *write_thread(void *arg) { const char *msg = "Writing to fd_b\n"; write(fd_b, msg, strlen(msg)); return NULL; }void *dup_thread(void *arg) { dup2(fd_a, fd_b); return NULL; }int main() { pthread_t writer, duper; fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644); fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644); pthread_create(&writer, NULL, write_thread, NULL); pthread_create(&duper, NULL, dup_thread, NULL); pthread_join(writer, NULL); pthread_join(duper, NULL); close(fd_a); close(fd_b); return 0; }--
Quoted 11 lines
If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via
mmap()orumem_reg.Looking at existing implementation:
- for mlx5e, it looks like it assumes the
set_group_asid()works only withoutDRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulatedYes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.
Quoted 16 lines
Quoted 2 lines
ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed withdma_unmapBut the issue I mention is that the, from the view of the vDPA bus:
1) it offers
set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is usedSo theoretically,
set_group_as_id()could happen between
dma_addr=dma_map();and
dma_unmap(dma_adrr);Quoted 7 lines
Quoted 1 line
But those two
dma_addrrefers to the different address space.I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?
Quoted 4 lines
Just to clarify, I for dma_map()/dma_unmap() this is not the UAPI part (and we don't have that). I basically mean the dma API which is used by virtio-vDPA.
So let's say that the
virtio_vdpadriver is able to call .set_group_asid just to explain how much it is protected:->vduse_dev_map_page and ->vduse_dev_unmap_page
In this series,rcu_dereferencemakes sure that only the stack copy of the domain pointer is used, but the next version the code will have aread_locktaken protecting thatset_group_asidactions ontoken.group->ascannot run at the same time that thevduse_domain_map_page(...)orvduse_domain_unmap_page(...)function call run in cpu0:static
dma_addr_tvduse_dev_map_page(unionvirtio_maptoken, struct page *page,
unsigned long offset,size_tsize, enumdma_data_directiondir, unsigned long attrs){ struct vduse_iova_domain *domain; dma_addr_t r; if (!token.group) return DMA_MAPPING_ERROR; read_lock(&dev->groups[group].as_lock); domain = token.group->as->domain; r = vduse_domain_map_page(domain, page, offset, size, dir, attrs); read_unlock(&dev->groups[group].as_lock); return r; }static void vduse_dev_unmap_page(union
virtio_maptoken,dma_addr_tdma_addr,
size_tsize, enumdma_data_directiondir, unsigned long attrs){ struct vduse_iova_domain *domain; if (!token.group) return; read_lock(&dev->groups[group].as_lock); domain = token.group->as->domain; vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs); read_unlock(&dev->groups[group].as_lock); }---
->
vduse_set_group_asidwill have the correspondingwrite_lock, so it cannot modify any vq group AS pointer while they're mapping or unmapping pages from a domain:
static void vduse_set_group_asid_nomsg(structvduse_dev*dev,
unsigned int group, unsigned int asid){ write_lock(&dev->groups[group].as_lock); dev->groups[group].as = &dev->as[asid]; write_unlock(&dev->groups[group].as_lock); }---
Now your question:
assuming group0.as = as0
cpu0]
dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1]set_group_asid(group0.as, as1)
cpu0]dma_unmap(group0.as, addr, DMA_FROM_DEVICE)cpu0 may read as1 while it wants as0 actually?
--Yes, it can happen even with the rwlock or any other synchronization in VDUSE, as it does not order the vdpa core calls.
This is the code I have in mind to prove this, let me know if I'm missing something from your scenario:
static int
thread1_func(void *data){ vduse_dev_map_page(token, ...); printk(KERN_INFO "map_page called\n"); // TODO: Need a way to know if the group ASID has changed to // know if this unmap is valid or not. vduse_dev_unmap_page(token, ...); printk(KERN_INFO "unmap_page called\n"); return 0; }static int
thread2_func(void *data){ u32 vq_group = 0; u32 asid = 1; vdpa->ops->set_group_asid(global_vdpa, vq_group, asid); printk(KERN_INFO "set_group_asid called\n"); return 0; }static int
__initkernel_threads_init(void){ struct task_struct *thread1, *thread2; thread1 = kthread_run(thread1_func, NULL, "vdpa_map_unmap"); thread2 = kthread_run(thread2_func, NULL, "vdpa_set_asid"); return 0; }---
Quoted 7 lines
I keep thinking that this is a race condition in the hypothetical vdpa core or the module that runs this code, not solvable in VDUSE. We need to add barriers at this level, not VDUSE.
But I think it is a good idea to add some code in VDUSE to mitigate it. Maybe just adding the expected vq queue index to
vdev->map->map_pageandvdev->map->unmap_page?
This seems to be more complicated than encoding asid to the iova.
Quoted 2 lines
This way we can issue
an error if the race happens.
As discussed, it could be avoided by disallowing the call to set_group_asid() when DRIVER_OK is set.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Instead
of trying to do synchronization, maybe we can simply failset_group_asidifDRIVER_OKis set.That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.
Quoted 5 lines
Quoted 3 lines
Did you mean the SVQ may update the group asid while
DRIVER_OKis set? If yes, we need to fix that.Not at this moment.
Great.
Thanks
Michael S. Tsirkin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
In reply to Eugenio Perez MartinQuoted 7 lines · Eugenio Perez Martin
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?
generally rcu itself does not need ONCE macros. these are for funky lockless things, and rcu can be seen as a kind of lock, after all.
Signature · 1 line
MST
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Eugenio Perez Martin
RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a
READ_ONCE/WRITE_ONCE, would that be more clear?
Quoted 4 lines
generally rcu itself does not need ONCE macros. these are for funky lockless things, and rcu can be seen as a kind of lock, after all.
Right, I meant to replace RCU by READ_ONCE / WRITE ONCE. But I also prefer RCU.
kernel test robot
build botRe: [PATCH v9 5/6] vduse: add vq group asid support
In reply to Eugenio Pérezkernel test robot noticed the following build warnings:
Hi Eugenio,
kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.18-rc5 next-20251113]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Eugenio-P-rez/vduse-add-v1-API-definition/20251113-200303 base: linus/master patch link: https://lore.kernel.org/r/20251113115558.1277981-6-eperezma%40redhat.com patch subject: [PATCH v9 5/6] vduse: add vq group asid support config: arc-randconfig-001-20251114 (https://download.01.org/0day-ci/archive/20251114/[email protected]/config) compiler: arc-linux-gcc (GCC) 8.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251114/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_sync_single_for_device':
>> drivers/vdpa/vdpa_user/vduse_dev.c:910:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_sync_single_for_cpu':
drivers/vdpa/vdpa_user/vduse_dev.c:929:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_map_page':
drivers/vdpa/vdpa_user/vduse_dev.c:949:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_unmap_page':
drivers/vdpa/vdpa_user/vduse_dev.c:971:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_alloc_coherent':
drivers/vdpa/vdpa_user/vduse_dev.c:989:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_need_sync':
drivers/vdpa/vdpa_user/vduse_dev.c:1034:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~
drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_max_mapping_size':
drivers/vdpa/vdpa_user/vduse_dev.c:1061:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
struct vduse_dev *vdev;
^~~~vim +/vdev +910 drivers/vdpa/vdpa_user/vduse_dev.c
c8a6153b6c59d9 Xie Yongji 2021-08-31 905
1c14b0e4ba9883 Jason Wang 2025-09-24 906 static void vduse_dev_sync_single_for_device(union virtio_map token,
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19 907 dma_addr_t dma_addr, size_t size,
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19 908 enum dma_data_direction dir)
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19 909 {
f98606c92d03d8 Eugenio Pérez 2025-11-13 @910 struct vduse_dev *vdev;
1e9ae6f6f2d50d Eugenio Pérez 2025-11-13 911 struct vduse_as *as;
f98606c92d03d8 Eugenio Pérez 2025-11-13 912 struct vduse_iova_domain *domain;
f98606c92d03d8 Eugenio Pérez 2025-11-13 913
f98606c92d03d8 Eugenio Pérez 2025-11-13 914 if (!token.group)
f98606c92d03d8 Eugenio Pérez 2025-11-13 915 return;
f98606c92d03d8 Eugenio Pérez 2025-11-13 916
f98606c92d03d8 Eugenio Pérez 2025-11-13 917 vdev = token.group->dev;
1e9ae6f6f2d50d Eugenio Pérez 2025-11-13 918 rcu_read_lock();
1e9ae6f6f2d50d Eugenio Pérez 2025-11-13 919 as = rcu_dereference(token.group->as);
1e9ae6f6f2d50d Eugenio Pérez 2025-11-13 920 domain = as->domain;
1e9ae6f6f2d50d Eugenio Pérez 2025-11-13 921 rcu_read_unlock();
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19 922 vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19 923 }
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19 924 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v9 5/6] vduse: add vq group asid support
In reply to Eugenio Pérezkernel test robot noticed the following build warnings:
Hi Eugenio,
kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.18-rc5 next-20251113]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Eugenio-P-rez/vduse-add-v1-API-definition/20251113-200303 base: linus/master patch link: https://lore.kernel.org/r/20251113115558.1277981-6-eperezma%40redhat.com patch subject: [PATCH v9 5/6] vduse: add vq group asid support config: i386-randconfig-062-20251114 (https://download.01.org/0day-ci/archive/20251114/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251114/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> drivers/vdpa/vdpa_user/vduse_dev.c:462:9: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:462:9: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:462:9: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:919:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:919:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:919:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:938:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:938:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:938:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:959:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:959:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:959:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:980:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:980:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:980:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:1001:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:1001:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:1001:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:1025:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:1025:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:1025:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:1044:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:1044:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:1044:14: sparse: struct vduse_as * drivers/vdpa/vdpa_user/vduse_dev.c:1071:14: sparse: sparse: incompatible types in comparison expression (different address spaces): drivers/vdpa/vdpa_user/vduse_dev.c:1071:14: sparse: struct vduse_as [noderef] __rcu * drivers/vdpa/vdpa_user/vduse_dev.c:1071:14: sparse: struct vduse_as * >> drivers/vdpa/vdpa_user/vduse_dev.c:1074:23: sparse: sparse: dereference of noderef expression
vim +462 drivers/vdpa/vdpa_user/vduse_dev.c
452
453 /* Force set the asid to a vq group without a message to the VDUSE device */
454 static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
455 unsigned int group, unsigned int asid)
456 {
457 /*
458 * Two concurrent updates to this pointer are valid as they cannot
459 * point to an invalid region. It is ok for them to race as long as
460 * the readers see a consistent state through RCU.
461 */
> 462 rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);
463 }
464
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v9 5/6 vduse: add vq group asid support
In reply to Eugenio PérezQuoted 409 lines
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected bydomain_lock.Acked-by Jason Wang <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.
v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.
v7:
* Take write lock in the error path (Jason).
v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.
v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.
v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).
v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)
v2:
* Convert the use of mutex to rwlock.
RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.
RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.
---
drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------include/uapi/linux/vduse.h| 53 ++++-2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +1
-0@@ -11,6 +11,7 @@ #include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h>@@ -41,6 +42,7 @@
#defineVDUSE_DEV_MAX(1U << MINORBITS) #defineVDUSE_DEV_MAX_GROUPS0xffff+#defineVDUSE_DEV_MAX_AS0xffff
#defineVDUSE_MAX_BOUNCE_SIZE(1024 * 1024 * 1024) #defineVDUSE_MIN_BOUNCE_SIZE(1024 * 1024) #defineVDUSE_BOUNCE_SIZE(64 * 1024 * 1024)drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -86,7 +88,14 @@ struct vduse_umem { struct mm_struct *mm; };drivers/vdpa/vdpa_user/vduse_dev.c +6
-0+struct vduse_as { + struct vduse_iova_domain *domain; + struct vduse_umem *umem; + struct mutex mem_lock; +}; +struct
vduse_vq_group{
+ structvduse_as*as__rcu;
structvduse_dev*dev;
};drivers/vdpa/vdpa_user/vduse_dev.c +2
-3@@ -94,7 +103,7 @@ struct vduse_dev { struct vduse_vdpa *vdev; struct device *dev; struct vduse_virtqueue **vqs; - struct vduse_iova_domain *domain; + struct vduse_as *as; char *name; struct mutex lock; spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev { u32 vq_num; u32 vq_align; u32 ngroups; - struct vduse_umem *umem; + u32 nas; struct vduse_vq_group *groups; - struct mutex mem_lock; unsigned int bounce_size; struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status) return vduse_dev_msg_sync(dev, &msg); }drivers/vdpa/vdpa_user/vduse_dev.c +1
-1-static int vduse_dev_update_iotlb(struct vduse_dev *dev, +static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,u64 start, u64 last) {
structvduse_dev_msgmsg = { 0 };drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev, return -EINVAL;
msg.req.type =VDUSE_UPDATE_IOTLB;
- msg.req.iova.start = start;
- msg.req.iova.last = last;
+ if (dev->api_version<VDUSE_API_VERSION_1) {
+ msg.req.iova.start = start;
+ msg.req.iova.last = last;
+ } else {
+ msg.req.iova_v2.start = start;
+ msg.req.iova_v2.last = last;
+ msg.req.iova_v2.asid = asid;
+ }
returnvduse_dev_msg_sync(dev, &msg); }drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) return mask; }drivers/vdpa/vdpa_user/vduse_dev.c +12
-0+/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, + unsigned int group, unsigned int asid) +{ + /* + * Two concurrent updates to this pointer are valid as they cannot + * point to an invalid region. It is ok for them to race as long as + * the readers see a consistent state through RCU. + */ + rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]); +} +static void
vduse_dev_reset(struct vduse_dev *dev)
{
int i;
- structvduse_iova_domain*domain =dev->domain;
/* The coherent mappings are handled invduse_dev_free_coherent()*/
- if (domain &&domain->bounce_map)
- vduse_domain_reset_bounce_map(domain);
+ for (i = 0; i <dev->nas; i++) {
+ structvduse_iova_domain*domain = dev->as[i].domain;drivers/vdpa/vdpa_user/vduse_dev.c +7
-0+ + if (domain && domain->bounce_map) + vduse_domain_reset_bounce_map(domain); + } + + for (i = 0; i < dev->ngroups; i++) + vduse_set_group_asid_nomsg(dev, i, 0);
down_write(&dev->rwsem);drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx) return ret; }drivers/vdpa/vdpa_user/vduse_dev.c +23
-0+static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, + unsigned int asid) +{ + struct vduse_dev *dev = vdpa_to_vduse(vdpa); + struct vduse_dev_msg msg = { 0 }; + int r; + + if (dev->api_version < VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) + return -EINVAL; + + msg.req.type = VDUSE_SET_VQ_GROUP_ASID; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r = vduse_dev_msg_sync(dev, &msg); + if (r < 0) + return r; + + vduse_set_group_asid_nomsg(dev, group, asid); + return 0; +} +static int
vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx, struct vdpa_vq_state *state){drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa, struct vduse_dev *dev = vdpa_to_vduse(vdpa); int ret;
- ret =vduse_domain_set_map(dev->domain, iotlb);
+ ret =vduse_domain_set_map(dev->as[asid].domain, iotlb); if (ret)
return ret;
- ret =vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+ ret =vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX); if (ret) {
- vduse_domain_clear_map(dev->domain, iotlb);
+ vduse_domain_clear_map(dev->as[asid].domain, iotlb); return ret;
}drivers/vdpa/vdpa_user/vduse_dev.c +2
-0@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) { struct vduse_dev *vdev; + struct vduse_as *as; struct vduse_iova_domain *domain;
if (!token.group)
return;
vdev =token.group->dev;
- domain =vdev->domain;
-
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();vduse_domain_sync_single_for_device(domain, dma_addr, size, dir); }drivers/vdpa/vdpa_user/vduse_dev.c +1
-0@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) { struct vduse_dev *vdev; + struct vduse_as *as; struct vduse_iova_domain *domain;
if (!token.group)
return;
vdev =token.group->dev;
- domain =vdev->domain;
-
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir); }drivers/vdpa/vdpa_user/vduse_dev.c +2
-0@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) { struct vduse_dev *vdev; + struct vduse_as *as; struct vduse_iova_domain *domain; + dma_addr_t r;
if (!token.group)
returnDMA_MAPPING_ERROR;
vdev =token.group->dev;
- domain =vdev->domain;
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();
+ r =vduse_domain_map_page(domain, page, offset, size, dir, attrs);
- returnvduse_domain_map_page(domain, page, offset, size, dir, attrs);
+ return r;
}
static void vduse_dev_unmap_page(unionvirtio_maptoken,dma_addr_tdma_addr,drivers/vdpa/vdpa_user/vduse_dev.c +1
-0@@ -901,21 +969,25 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr, unsigned long attrs) { struct vduse_dev *vdev; + struct vduse_as *as; struct vduse_iova_domain *domain;
if (!token.group)
return;
vdev =token.group->dev;
- domain =vdev->domain;
-
- returnvduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();
+ vduse_domain_unmap_page(domain,dma_addr, size, dir, attrs); }
static void *vduse_dev_alloc_coherent(unionvirtio_maptoken,size_tsize,dma_addr_t*dma_addr,gfp_tflag) {
structvduse_dev*vdev;
+ structvduse_as*as;
structvduse_iova_domain*domain;
unsigned long iova;
void *addr;drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -925,13 +997,14 @@ static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size, return NULL;
vdev =token.group->dev;
- domain =vdev->domain;
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();
addr =vduse_domain_alloc_coherent(domain, size, (dma_addr_t *)&iova, flag);
- if (!addr)
- return NULL;
-
- *dma_addr = (dma_addr_t)iova;
+ if (addr)
+ *dma_addr = (dma_addr_t)iova;
return addr;
}drivers/vdpa/vdpa_user/vduse_dev.c +1
-0@@ -941,29 +1014,39 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size, unsigned long attrs) { struct vduse_dev *vdev; + struct vduse_as *as; struct vduse_iova_domain *domain;
if (!token.group)
return;
vdev =token.group->dev;
- domain =vdev->domain;
-
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();
vduse_domain_free_coherent(domain, size, vaddr,dma_addr, attrs);
+ mutex_unlock(&vdev->domain_lock);
}
static boolvduse_dev_need_sync(union virtio_map token, dma_addr_t dma_addr){
structvduse_dev*vdev;
+ structvduse_as*as;
structvduse_iova_domain*domain;
+ size_tbounce_size;
if (!token.group)
return false;
vdev =token.group->dev;
- domain =vdev->domain;
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();
+ bounce_size =domain->bounce_size;
- returndma_addr<domain->bounce_size;
+ returndma_addr<bounce_size;
}
static intvduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)drivers/vdpa/vdpa_user/vduse_dev.c +2
-0@@ -976,15 +1059,22 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr) static size_t vduse_dev_max_mapping_size(union virtio_map token) { struct vduse_dev *vdev; + struct vduse_as *as; struct vduse_iova_domain *domain; + size_t bounce_size;
if (!token.group)
return 0;
Quoted 8 lines
vdev =token.group->dev;
- domain =vdev->domain;
+ rcu_read_lock();
+ as =rcu_dereference(token.group->as);
+ domain =as->domain;
+ rcu_read_unlock();
+ domain =token.group->as->domain;
This line should be removed?
Thanks,
Yongji
Eugenio Perez Martin
maintainerRe: [PATCH v9 5/6] vduse: add vq group asid support
Quoted 411 lines · 谢永吉
Quoted 411 lines
Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by
domain_lock.Acked-by Jason Wang <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>---
v9:
* Replace mutex with RCU, as the vdpamap_opscan run from atomic context.v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.v7:
* Take write lock in the error path (Jason).v6:
* Makevdpa_dev_adduse gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version <VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.v5:
* Properly return errno ifcopy_to_userreturns >0 inVDUSE_IOTLB_GET_FDioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members invduse_dev_request.v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Considerconfig->nas== 0 as a fail (Jason).v3:
* Get the vduse domain through thevduse_asin the map functions (Jason).
* Squash with the patch creating thevduse_asstruct (Jason).
* CreateVDUSE_DEV_MAX_ASinstead of comparing agains a magic number (Jason)v2:
* Convert the use of mutex to rwlock.RFC v3:
* IncreaseVDUSE_MAX_VQ_GROUPSto0xffff(Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about mergingVDUSE_IOTLB_GET_FDioctl withVDUSE_IOTLB_GET_INFO.
* Use ofarray_index_nospecin VDUSE device ioctls.
* Embedvduse_iotlb_entryintovduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.RFC v2:
* Make iotlb entry the last one ofvduse_iotlb_entry_v2so the first part of the struct is the same.--- drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h | 53 ++++- 2 files changed, 314 insertions(+), 109 deletions(-)drivers/vdpa/vdpa_user/vduse_dev.c +115
-30@@ -11,6 +11,7 @@#include "linux/virtio_net.h" #include <linux/init.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/cdev.h> #include <linux/device.h> #include <linux/eventfd.h> @@ -41,6 +42,7 @@#define VDUSE_DEV_MAX (1U << MINORBITS)#define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)@@ -86,7 +88,14 @@ struct vduse_umem {struct mm_struct *mm; }; +struct vduse_as { +struct vduse_iova_domain *domain; +struct vduse_umem *umem; +struct mutex mem_lock; +}; +struct vduse_vq_group { +struct vduse_as *as__rcu ;struct vduse_dev *dev; };@@ -94,7 +103,7 @@ struct vduse_dev {struct vduse_vdpa *vdev;struct device *dev;struct vduse_virtqueue **vqs; -struct vduse_iova_domain *domain; +struct vduse_as *as;char *name;struct mutex lock;spinlock_t msg_lock;@@ -122,9 +131,8 @@ struct vduse_dev {u32 vq_num;u32 vq_align;u32 ngroups; -struct vduse_umem *umem; +u32 nas;struct vduse_vq_group *groups; -struct mutex mem_lock;unsigned int bounce_size;struct mutex domain_lock; };@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)return vduse_dev_msg_sync (dev, &msg); } -static int vduse_dev_update_iotlb (struct vduse_dev *dev, +static int vduse_dev_update_iotlb (struct vduse_dev *dev,u32 asid,u64 start,u64 last) {struct vduse_dev_msg msg = {0 };@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,return -EINVAL ; msg.req.type =VDUSE_UPDATE_IOTLB ; - msg.req.iova.start = start; - msg.req.iova.last = last; +if (dev->api_version <VDUSE_API_VERSION_1 ) { + msg.req.iova.start = start; + msg.req.iova.last = last; + }else { + msg.req.iova_v2.start = start; + msg.req.iova_v2.last = last; + msg.req.iova_v2.asid = asid; + }return vduse_dev_msg_sync (dev, &msg); }@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)return mask; } +/* Force set the asid to a vq group without a message to the VDUSE device */ +static void vduse_set_group_asid_nomsg (struct vduse_dev *dev, +unsigned int group,unsigned int asid) +{ +/* +* Two concurrent updates to this pointer are valid as they cannot +* point to an invalid region. It is ok for them to race as long as +* the readers see a consistent state through RCU. +*/ +rcu_assign_pointer (dev->groups[group].as, &dev->as[asid]); +} +static void vduse_dev_reset (struct vduse_dev *dev) {int i; -struct vduse_iova_domain *domain = dev->domain;/* The coherent mappings are handled in vduse_dev_free_coherent() */ -if (domain && domain->bounce_map) -vduse_domain_reset_bounce_map (domain); +for (i =0 ; i < dev->nas; i++) { +struct vduse_iova_domain *domain = dev->as[i].domain; + +if (domain && domain->bounce_map) +vduse_domain_reset_bounce_map (domain); + } + +for (i =0 ; i < dev->ngroups; i++) +vduse_set_group_asid_nomsg (dev, i,0 );down_write (&dev->rwsem);@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)return ret; } +static int vduse_set_group_asid (struct vdpa_device *vdpa,unsigned int group, +unsigned int asid) +{ +struct vduse_dev *dev =vdpa_to_vduse (vdpa); +struct vduse_dev_msg msg = {0 }; +int r; + +if (dev->api_version <VDUSE_API_VERSION_1 || + group >= dev->ngroups || asid >= dev->nas) +return -EINVAL ; + + msg.req.type =VDUSE_SET_VQ_GROUP_ASID ; + msg.req.vq_group_asid.group = group; + msg.req.vq_group_asid.asid = asid; + + r =vduse_dev_msg_sync (dev, &msg); +if (r <0 ) +return r; + +vduse_set_group_asid_nomsg (dev, group, asid); +return 0 ; +} +static int vduse_vdpa_get_vq_state (struct vdpa_device *vdpa,u16 idx,struct vdpa_vq_state *state) {@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret; - ret =vduse_domain_set_map (dev->domain, iotlb); + ret =vduse_domain_set_map (dev->as[asid].domain, iotlb);if (ret)return ret; - ret =vduse_dev_update_iotlb (dev,0ULL ,ULLONG_MAX ); + ret =vduse_dev_update_iotlb (dev, asid,0ULL ,ULLONG_MAX );if (ret) { -vduse_domain_clear_map (dev->domain, iotlb); +vduse_domain_clear_map (dev->as[asid].domain, iotlb);return ret; }@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .get_vq_affinity = vduse_vdpa_get_vq_affinity, .reset = vduse_vdpa_reset, .set_map = vduse_vdpa_set_map, + .set_group_asid = vduse_set_group_asid, .get_vq_map = vduse_get_vq_map, .free = vduse_vdpa_free, };@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_device (domain, dma_addr, size, dir); }@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token, enum dma_data_direction dir) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); }@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +dma_addr_t r;if (!token.group)return DMA_MAPPING_ERROR ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + r =vduse_domain_map_page (domain, page, offset, size, dir, attrs); -return vduse_domain_map_page (domain, page, offset, size, dir, attrs); +return r; } static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,@@ -901,21 +969,25 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - -return vduse_domain_unmap_page (domain, dma_addr, size, dir, attrs); +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); +vduse_domain_unmap_page (domain, dma_addr, size, dir, attrs); }static void *vduse_dev_alloc_coherent (union virtio_map token,size_t size,dma_addr_t *dma_addr,gfp_t flag) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;unsigned long iova;void *addr;@@ -925,13 +997,14 @@ static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,return NULL ; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); addr =vduse_domain_alloc_coherent (domain, size, (dma_addr_t *)&iova, flag); -if (!addr) -return NULL ; - - *dma_addr = (dma_addr_t )iova; +if (addr) + *dma_addr = (dma_addr_t )iova;return addr; }@@ -941,29 +1014,39 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size, unsigned long attrs) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain;if (!token.group)return ; vdev = token.group->dev; - domain = vdev->domain; - +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock ();vduse_domain_free_coherent (domain, size, vaddr, dma_addr, attrs); +mutex_unlock (&vdev->domain_lock); }static bool vduse_dev_need_sync (union virtio_map token,dma_addr_t dma_addr) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +size_t bounce_size;if (!token.group)return false; vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + bounce_size = domain->bounce_size; -return dma_addr < domain->bounce_size; +return dma_addr < bounce_size; }static int vduse_dev_mapping_error (union virtio_map token,dma_addr_t dma_addr)@@ -976,15 +1059,22 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)static size_t vduse_dev_max_mapping_size (union virtio_map token) {struct vduse_dev *vdev; +struct vduse_as *as;struct vduse_iova_domain *domain; +size_t bounce_size;if (!token.group)return 0 ;
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr) · line 1074
Quoted 7 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5
-1vdev = token.group->dev; - domain = vdev->domain; +rcu_read_lock (); + as =rcu_dereference (token.group->as); + domain = as->domain; +rcu_read_unlock (); + domain = token.group->as->domain;This line should be removed?
Totally, thank you very much for the catch!
Eugenio Pérez
maintainerPATCH v9 6/6 vduse: bump version number
In reply to Eugenio PérezFinalize the series by advertising VDUSE API v1 support to userspace.
Now that all required infrastructure for v1 (ASIDs, VQ groups, update_iotlb_v2) is in place, VDUSE devices can opt in to the new features.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +2 -2
@@ -2188,7 +2188,7 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,break ; ret = -EINVAL ; -if (api_version >VDUSE_API_VERSION ) +if (api_version >VDUSE_API_VERSION_1 )break ; ret =0 ;
@@ -2255,7 +2255,7 @@ static int vduse_open(struct inode *inode, struct file *file)if (!control)return -ENOMEM ; - control->api_version =VDUSE_API_VERSION ; + control->api_version =VDUSE_API_VERSION_1 ; file->private_data = control;return 0 ;
--
2.51.1