Skip to contentSkip to conversation

Add multiple address spaces support to VDUSE

PATCH Superseded Eugenio Pérez 5 participants 29 messages Nov 13, 2025 – Dec 8, 2025 virtio core

v9 of 15 Superseded Latest is v15 →

lore
Share & tools
lore

Eugenio Pérez

author maintainer
#0 lore raw

v9 cover Add multiple address spaces support to VDUSE

When used by vhost-vDPA bus driver for VM, the control virtqueue should be shadowed via userspace VMM (QEMU) instead of being assigned directly to Guest. This is because QEMU needs to know the device state in order to start and stop device correctly (e.g for Live Migration).

This requies to isolate the memory mapping for control virtqueue presented by vhost-vDPA to prevent guest from accessing it directly.

This series add support to multiple address spaces in VDUSE device allowing selective virtqueue isolation through address space IDs (ASID).

The VDUSE device needs to report:
* Number of virtqueue groups
* Association of each vq group with each virtqueue
* Number of address spaces supported.

Then, the vDPA driver can modify the ASID assigned to each VQ group to isolate the memory AS. This aligns VDUSE with vdpa_sim and nvidia mlx5 devices which already support ASID.

This helps to isolate the environments for the virtqueues that will not be assigned directly. E.g in the case of virtio-net, the control virtqueue will not be assigned directly to guest.

Also, to be able to test this patch, the user needs to manually revert 56e71885b034 ("vduse: Temporarily fail if control queue feature requested").

Tested by creating a VDUSE device OVS with and without MQ, and live migrating between two hosts back and forth while maintaining ping alive in all the stages. All tested with and without lockdep.

PATCH v9:
* Change to RCU.

PATCH v8:
* Revert the change from mutex to rwlock (MST).

PATCH v7:
* Fix not taking the write lock in the registering vdpa device error path (Jason).

PATCH v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ in group and nas handling at device creation (MST).
* Fix struct name not matching in the doc.
* s/sepparate/separate (MST).

PATCH v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Revert core vdpa changes (Jason).
* Fix group == ngroup case in checking VQ_SETUP argument (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

PATCH v4:
* Consider config->nas == 0 and config->ngroups == 0 as a fail (Jason).
* Revert the "invalid vq group" concept and assume 0 if not set.
* Divide each domain bounce size between the device bounce size (Jason).
* Revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).

PATCH v3:
* Make the default group an invalid group as long as VDUSE device does not set it to some valid u32 value. Modify the vdpa core to take that into account (Jason). Adapt all the virtio_map_ops callbacks to it.
* Make setting status DRIVER_OK fail if vq group is not valid.
* Create the VDUSE_DEV_MAX_GROUPS and VDUSE_DEV_MAX_AS instead of using a magic number
* Remove the _int name suffix from struct vduse_vq_group.
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash the patch implementing the AS logic with the patch creating the vduse_as struct (Jason).

PATCH v2:
* Now the vq group is in vduse_vq_config struct instead of issuing one VDUSE message per vq.
* Convert the use of mutex to rwlock (Xie Yongji).

PATCH v1:
* Fix: Remove BIT_ULL(VIRTIO_S_*), as _S_ is already the bit (Maxime)
* Using vduse_vq_group_int directly instead of an empty struct in union virtio_map.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs. Also, it's a dynamic array now. Same with ASID.
* Move the valid vq groups range check to vduse_validate_config.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Use of array_index_nospec in VDUSE device ioctls.
* Move the umem mutex to asid struct so there is no contention between ASIDs.
* Remove the descs vq group capability as it will not be used and we can add it on top.
* Do not ask for vq groups in number of vq groups < 2.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.

RFC v2:
* Cache group information in kernel, as we need to provide the vq map tokens properly.
* Add descs vq group to optimize SVQ forwarding and support indirect descriptors out of the box.
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.
* Fixes detected testing with OVS+VDUSE.

Eugenio Pérez (6):
vduse: add v1 API definition
vduse: add vq group support
vduse: return internal vq group struct as map token
vduse: refactor vdpa_dev_add for goto err handling
vduse: add vq group asid support
vduse: bump version number

 drivers/vdpa/vdpa_user/vduse_dev.c | 496 +++++++++++++++++++++++------
 include/linux/virtio.h             |   6 +-
 include/uapi/linux/vduse.h         |  67 +++-
 3 files changed, 455 insertions(+), 114 deletions(-)
Signature · 1 line
2.51.1
author maintainer
#1 lore raw

PATCH v9 1/6 vduse: add v1 API definition

This allows the kernel to detect whether the userspace VDUSE device supports the VQ group and ASID features. VDUSE devices that don't set the V1 API will not receive the new messages, and vdpa device will be created with only one vq group and asid.

The next patches implement the new feature incrementally, only enabling the VDUSE device to set the V1 API version by the end of the series.

---
 include/uapi/linux/vduse.h | 4 ++++
 1 file changed, 4 insertions(+)
include/uapi/linux/vduse.h +4 -0
index 10ad71aa00d6..ccb92a1efce0 100644
@@ -10,6 +10,10 @@
 
 #define VDUSE_API_VERSION	0
 
+/* VQ groups and ASID support */
+
+#define VDUSE_API_VERSION_1	1
+
 /*
  * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
  * This is used for future extension.
Signature · 1 line
2.51.1
author maintainer
#2 lore raw

PATCH v9 2/6 vduse: add vq group support

In reply to Eugenio Pérez

This allows separate the different virtqueues in groups that shares the same address space. Asking the VDUSE device for the groups of the vq at the beginning as they're needed for the DMA API.

Allocating 3 vq groups as net is the device that need the most groups:
* Dataplane (guest passthrough)
* CVQ
* Shadowed vrings.

Future versions of the series can include dynamic allocation of the groups array so VDUSE can declare more groups.

---
v6:
* s/sepparate/separate (MST).
* s/dev->api_version < 1/dev->api_version < VDUSE_API_VERSION_1

v5:
* Revert core vdpa changes (Jason).
* Fix group == ngroup case in checking VQ_SETUP argument (Jason).

v4:
* Revert the "invalid vq group" concept and assume 0 if not set (Jason).
* Make config->ngroups == 0 invalid (Jason).

v3:
* Make the default group an invalid group as long as VDUSE device does not set it to some valid u32 value. Modify the vdpa core to take that into account (Jason).
* Create the VDUSE_DEV_MAX_GROUPS instead of using a magic number

v2:
* Now the vq group is in vduse_vq_config struct instead of issuing one VDUSE message per vq.

v1:
* Fix: Remove BIT_ULL(VIRTIO_S_*), as _S_ is already the bit (Maxime)

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove the descs vq group capability as it will not be used and we can add it on top.
* Do not ask for vq groups in number of vq groups < 2.
* Move the valid vq groups range check to vduse_validate_config.

RFC v2:
* Cache group information in kernel, as we need to provide the vq map tokens properly.
* Add descs vq group to optimize SVQ forwarding and support indirect descriptors out of the box.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 48 ++++++++++++++++++++++++++----
 include/uapi/linux/vduse.h         | 12 ++++++--
 2 files changed, 52 insertions(+), 8 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +43 -5
index e7bced0b5542..d5c90c599698 100644
@@ -39,6 +39,7 @@
 #define DRV_LICENSE  "GPL v2"
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
+#define VDUSE_DEV_MAX_GROUPS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -58,6 +59,7 @@ struct vduse_virtqueue {
 	struct vdpa_vq_state state;
 	bool ready;
 	bool kicked;
+	u32 vq_group;
 	spinlock_t kick_lock;
 	spinlock_t irq_lock;
 	struct eventfd_ctx *kickfd;
@@ -114,6 +116,7 @@ struct vduse_dev {
 	u8 status;
 	u32 vq_num;
 	u32 vq_align;
+	u32 ngroups;
 	struct vduse_umem *umem;
 	struct mutex mem_lock;
 	unsigned int bounce_size;
@@ -455,6 +458,7 @@ static void vduse_dev_reset(struct vduse_dev *dev)
 		vq->driver_addr = 0;
 		vq->device_addr = 0;
 		vq->num = 0;
+		vq->vq_group = 0;
 		memset(&vq->state, 0, sizeof(vq->state));
 
 		spin_lock(&vq->kick_lock);
@@ -592,6 +596,16 @@ static int vduse_vdpa_set_vq_state(struct vdpa_device *vdpa, u16 idx,
 	return 0;
 }
 
+static u32 vduse_get_vq_group(struct vdpa_device *vdpa, u16 idx)
+{
+	struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+
+	if (dev->api_version < VDUSE_API_VERSION_1)
+		return 0;
+
+	return dev->vqs[idx]->vq_group;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
 				struct vdpa_vq_state *state)
 {
@@ -789,6 +803,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
 	.set_vq_cb		= vduse_vdpa_set_vq_cb,
 	.set_vq_num             = vduse_vdpa_set_vq_num,
 	.get_vq_size		= vduse_vdpa_get_vq_size,
+	.get_vq_group		= vduse_get_vq_group,
 	.set_vq_ready		= vduse_vdpa_set_vq_ready,
 	.get_vq_ready		= vduse_vdpa_get_vq_ready,
 	.set_vq_state		= vduse_vdpa_set_vq_state,
@@ -1252,12 +1267,24 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 		if (config.index >= dev->vq_num)
 			break;
 
-		if (!is_mem_zero((const char *)config.reserved,
-				 sizeof(config.reserved)))
+		if (dev->api_version < VDUSE_API_VERSION_1 && config.group)
+			break;
+
+		if (dev->api_version >= VDUSE_API_VERSION_1) {
+			if (config.group >= dev->ngroups)
+				break;
+			if (dev->status & VIRTIO_CONFIG_S_DRIVER_OK)
+				break;
+		}
+
+		if (config.reserved1 ||
+		    !is_mem_zero((const char *)config.reserved2,
+				 sizeof(config.reserved2)))
 			break;
 
 		index = array_index_nospec(config.index, dev->vq_num);
 		dev->vqs[index]->num_max = config.max_size;
+		dev->vqs[index]->vq_group = config.group;
 		ret = 0;
 		break;
 	}
@@ -1737,12 +1764,20 @@ static bool features_is_valid(struct vduse_dev_config *config)
 	return true;
 }
 
-static bool vduse_validate_config(struct vduse_dev_config *config)
+static bool vduse_validate_config(struct vduse_dev_config *config,
+				  u64 api_version)
 {
 	if (!is_mem_zero((const char *)config->reserved,
 			 sizeof(config->reserved)))
 		return false;
 
+	if (api_version < VDUSE_API_VERSION_1 && config->ngroups)
+		return false;
+
+	if (api_version >= VDUSE_API_VERSION_1 &&
+	    (!config->ngroups || config->ngroups > VDUSE_DEV_MAX_GROUPS))
+		return false;
+
 	if (config->vq_align > PAGE_SIZE)
 		return false;
 
@@ -1858,6 +1893,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	dev->device_features = config->features;
 	dev->device_id = config->device_id;
 	dev->vendor_id = config->vendor_id;
+	dev->ngroups = (dev->api_version < VDUSE_API_VERSION_1)
+		       ? 1
+		       : config->ngroups;
 	dev->name = kstrdup(config->name, GFP_KERNEL);
 	if (!dev->name)
 		goto err_str;
@@ -1936,7 +1974,7 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
 			break;
 
 		ret = -EINVAL;
-		if (vduse_validate_config(&config) == false)
+		if (!vduse_validate_config(&config, control->api_version))
 			break;
 
 		buf = vmemdup_user(argp + size, config.config_size);
@@ -2017,7 +2055,7 @@ static int vduse_dev_init_vdpa(struct vduse_dev *dev, const char *name)
 
 	vdev = vdpa_alloc_device(struct vduse_vdpa, vdpa, dev->dev,
 				 &vduse_vdpa_config_ops, &vduse_map_ops,
-				 1, 1, name, true);
+				 dev->ngroups, 1, name, true);
 	if (IS_ERR(vdev))
 		return PTR_ERR(vdev);
 
include/uapi/linux/vduse.h +9 -3
index ccb92a1efce0..a3d51cf6df3a 100644
@@ -31,6 +31,7 @@
  * @features: virtio features
  * @vq_num: the number of virtqueues
  * @vq_align: the allocation alignment of virtqueue's metadata
+ * @ngroups: number of vq groups that VDUSE device declares
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space
@@ -45,7 +46,8 @@ struct vduse_dev_config {
 	__u64 features;
 	__u32 vq_num;
 	__u32 vq_align;
-	__u32 reserved[13];
+	__u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
+	__u32 reserved[12];
 	__u32 config_size;
 	__u8 config[];
 };
@@ -122,14 +124,18 @@ struct vduse_config_data {
  * struct vduse_vq_config - basic configuration of a virtqueue
  * @index: virtqueue index
  * @max_size: the max size of virtqueue
- * @reserved: for future use, needs to be initialized to zero
+ * @reserved1: for future use, needs to be initialized to zero
+ * @group: virtqueue group
+ * @reserved2: for future use, needs to be initialized to zero
  *
  * Structure used by VDUSE_VQ_SETUP ioctl to setup a virtqueue.
  */
 struct vduse_vq_config {
 	__u32 index;
 	__u16 max_size;
-	__u16 reserved[13];
+	__u16 reserved1;
+	__u32 group;
+	__u16 reserved2[10];
 };
 
 /*
Signature · 1 line
2.51.1
author maintainer
#3 lore raw

PATCH v9 3/6 vduse: return internal vq group struct as map token

In reply to Eugenio Pérez

Return the internal struct that represents the vq group as virtqueue map token, instead of the device. This allows the map functions to access the information per group.

At this moment all the virtqueues share the same vq group, that only can point to ASID 0. This change prepares the infrastructure for actual per-group address space handling

---
v4:
* Revert the "invalid vq group" concept, and assume 0 by default.
* Revert unnecesary blank line addition (Jason)

v3:
* Adapt all virtio_map_ops callbacks to handle empty tokens in case of invalid groups.
* Make setting status DRIVER_OK fail if vq group is not valid.
* Remove the _int name suffix from struct vduse_vq_group.

RFC v3:
* Make the vq groups a dynamic array to support an arbitrary number of them.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 100 ++++++++++++++++++++++++++---
 include/linux/virtio.h             |   6 +-
 2 files changed, 94 insertions(+), 12 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +91 -9
index d5c90c599698..b01f55a1c22d 100644
@@ -22,6 +22,7 @@
 #include <linux/uio.h>
 #include <linux/vdpa.h>
 #include <linux/nospec.h>
+#include <linux/virtio.h>
 #include <linux/vmalloc.h>
 #include <linux/sched/mm.h>
 #include <uapi/linux/vduse.h>
@@ -85,6 +86,10 @@ struct vduse_umem {
 	struct mm_struct *mm;
 };
 
+struct vduse_vq_group {
+	struct vduse_dev *dev;
+};
+
 struct vduse_dev {
 	struct vduse_vdpa *vdev;
 	struct device *dev;
@@ -118,6 +123,7 @@ struct vduse_dev {
 	u32 vq_align;
 	u32 ngroups;
 	struct vduse_umem *umem;
+	struct vduse_vq_group *groups;
 	struct mutex mem_lock;
 	unsigned int bounce_size;
 	struct mutex domain_lock;
@@ -606,6 +612,17 @@ static u32 vduse_get_vq_group(struct vdpa_device *vdpa, u16 idx)
 	return dev->vqs[idx]->vq_group;
 }
 
+static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
+{
+	struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+	u32 vq_group = vduse_get_vq_group(vdpa, idx);
+	union virtio_map ret = {
+		.group = &dev->groups[vq_group],
+	};
+
+	return ret;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
 				struct vdpa_vq_state *state)
 {
@@ -826,6 +843,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
 	.get_vq_affinity	= vduse_vdpa_get_vq_affinity,
 	.reset			= vduse_vdpa_reset,
 	.set_map		= vduse_vdpa_set_map,
+	.get_vq_map		= vduse_get_vq_map,
 	.free			= vduse_vdpa_free,
 };
 
@@ -833,7 +851,14 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
 					     dma_addr_t dma_addr, size_t size,
 					     enum dma_data_direction dir)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);
 }
@@ -842,7 +867,14 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
 					     dma_addr_t dma_addr, size_t size,
 					     enum dma_data_direction dir)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
@@ -852,7 +884,14 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
 				     enum dma_data_direction dir,
 				     unsigned long attrs)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return DMA_MAPPING_ERROR;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	return vduse_domain_map_page(domain, page, offset, size, dir, attrs);
 }
@@ -861,7 +900,14 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
 				 size_t size, enum dma_data_direction dir,
 				 unsigned long attrs)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	return vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
 }
@@ -869,11 +915,17 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
 static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,
 				      dma_addr_t *dma_addr, gfp_t flag)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
 	unsigned long iova;
 	void *addr;
 
 	*dma_addr = DMA_MAPPING_ERROR;
+	if (!token.group)
+		return NULL;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 	addr = vduse_domain_alloc_coherent(domain, size,
 					   (dma_addr_t *)&iova, flag);
 	if (!addr)
@@ -888,14 +940,28 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,
 				    void *vaddr, dma_addr_t dma_addr,
 				    unsigned long attrs)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	vduse_domain_free_coherent(domain, size, vaddr, dma_addr, attrs);
 }
 
 static bool vduse_dev_need_sync(union virtio_map token, dma_addr_t dma_addr)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return false;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	return dma_addr < domain->bounce_size;
 }
@@ -909,7 +975,14 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
 
 static size_t vduse_dev_max_mapping_size(union virtio_map token)
 {
-	struct vduse_iova_domain *domain = token.iova_domain;
+	struct vduse_dev *vdev;
+	struct vduse_iova_domain *domain;
+
+	if (!token.group)
+		return 0;
+
+	vdev = token.group->dev;
+	domain = vdev->domain;
 
 	return domain->bounce_size;
 }
@@ -1727,6 +1800,7 @@ static int vduse_destroy_dev(char *name)
 	if (dev->domain)
 		vduse_domain_destroy(dev->domain);
 	kfree(dev->name);
+	kfree(dev->groups);
 	vduse_dev_destroy(dev);
 	module_put(THIS_MODULE);
 
@@ -1896,6 +1970,13 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	dev->ngroups = (dev->api_version < VDUSE_API_VERSION_1)
 		       ? 1
 		       : config->ngroups;
+	dev->groups = kcalloc(dev->ngroups, sizeof(dev->groups[0]),
+			      GFP_KERNEL);
+	if (!dev->groups)
+		goto err_vq_groups;
+	for (u32 i = 0; i < dev->ngroups; ++i)
+		dev->groups[i].dev = dev;
+
 	dev->name = kstrdup(config->name, GFP_KERNEL);
 	if (!dev->name)
 		goto err_str;
@@ -1932,6 +2013,8 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 err_idr:
 	kfree(dev->name);
 err_str:
+	kfree(dev->groups);
+err_vq_groups:
 	vduse_dev_destroy(dev);
 err:
 	return ret;
@@ -2093,7 +2176,6 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 		return -ENOMEM;
 	}
 
-	dev->vdev->vdpa.vmap.iova_domain = dev->domain;
 	ret = _vdpa_register_device(&dev->vdev->vdpa, dev->vq_num);
 	if (ret) {
 		put_device(&dev->vdev->vdpa.dev);
include/linux/virtio.h +3 -3
index 96c66126c074..302109029700 100644
@@ -41,13 +41,13 @@ struct virtqueue {
 	void *priv;
 };
 
-struct vduse_iova_domain;
+struct vduse_vq_group;
 
 union virtio_map {
 	/* Device that performs DMA */
 	struct device *dma_dev;
-	/* VDUSE specific mapping data */
-	struct vduse_iova_domain *iova_domain;
+	/* VDUSE specific virtqueue group for doing map */
+	struct vduse_vq_group *group;
 };
 
 int virtqueue_add_outbuf(struct virtqueue *vq,
Signature · 1 line
2.51.1
author maintainer
#4 lore raw

PATCH v9 4/6 vduse: refactor vdpa_dev_add for goto err handling

In reply to Eugenio Pérez

Next patches introduce more error paths in this function. Refactor it so they can be accomodated through gotos.

---
v6: New in v6.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +14 -8
index b01f55a1c22d..97be04f73fbf 100644
@@ -2172,21 +2172,27 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 						  dev->bounce_size);
 	mutex_unlock(&dev->domain_lock);
 	if (!dev->domain) {
-		put_device(&dev->vdev->vdpa.dev);
-		return -ENOMEM;
+		ret = -ENOMEM;
+		goto domain_err;
 	}
 
 	ret = _vdpa_register_device(&dev->vdev->vdpa, dev->vq_num);
 	if (ret) {
-		put_device(&dev->vdev->vdpa.dev);
-		mutex_lock(&dev->domain_lock);
-		vduse_domain_destroy(dev->domain);
-		dev->domain = NULL;
-		mutex_unlock(&dev->domain_lock);
-		return ret;
+		goto register_err;
 	}
 
 	return 0;
+
+register_err:
+	mutex_lock(&dev->domain_lock);
+	vduse_domain_destroy(dev->domain);
+	dev->domain = NULL;
+	mutex_unlock(&dev->domain_lock);
+
+domain_err:
+	put_device(&dev->vdev->vdpa.dev);
+
+	return ret;
 }
 
 static void vdpa_dev_del(struct vdpa_mgmt_dev *mdev, struct vdpa_device *dev)
Signature · 1 line
2.51.1
author maintainer
#5 lore raw

PATCH v9 5/6 vduse: add vq group asid support

In reply to Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +264 -106
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
 	struct mm_struct *mm;
 };
 
+struct vduse_as {
+	struct vduse_iova_domain *domain;
+	struct vduse_umem *umem;
+	struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+	struct vduse_as *as __rcu;
 	struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
 	struct vduse_vdpa *vdev;
 	struct device *dev;
 	struct vduse_virtqueue **vqs;
-	struct vduse_iova_domain *domain;
+	struct vduse_as *as;
 	char *name;
 	struct mutex lock;
 	spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
 	u32 vq_num;
 	u32 vq_align;
 	u32 ngroups;
-	struct vduse_umem *umem;
+	u32 nas;
 	struct vduse_vq_group *groups;
-	struct mutex mem_lock;
 	unsigned int bounce_size;
 	struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
 	return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
 				  u64 start, u64 last)
 {
 	struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
 		return -EINVAL;
 
 	msg.req.type = VDUSE_UPDATE_IOTLB;
-	msg.req.iova.start = start;
-	msg.req.iova.last = last;
+	if (dev->api_version < VDUSE_API_VERSION_1) {
+		msg.req.iova.start = start;
+		msg.req.iova.last = last;
+	} else {
+		msg.req.iova_v2.start = start;
+		msg.req.iova_v2.last = last;
+		msg.req.iova_v2.asid = asid;
+	}
 
 	return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
 	return mask;
 }
 
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+				       unsigned int group, unsigned int asid)
+{
+	/*
+	 * Two concurrent updates to this pointer are valid as they cannot
+	 * point to an invalid region.  It is ok for them to race as long as
+	 * the readers see a consistent state through RCU.
+	 */
+	rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
 	int i;
-	struct vduse_iova_domain *domain = dev->domain;
 
 	/* The coherent mappings are handled in vduse_dev_free_coherent() */
-	if (domain && domain->bounce_map)
-		vduse_domain_reset_bounce_map(domain);
+	for (i = 0; i < dev->nas; i++) {
+		struct vduse_iova_domain *domain = dev->as[i].domain;
+
+		if (domain && domain->bounce_map)
+			vduse_domain_reset_bounce_map(domain);
+	}
+
+	for (i = 0; i < dev->ngroups; i++)
+		vduse_set_group_asid_nomsg(dev, i, 0);
 
 	down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
 	return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+				unsigned int asid)
+{
+	struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+	struct vduse_dev_msg msg = { 0 };
+	int r;
+
+	if (dev->api_version < VDUSE_API_VERSION_1 ||
+	    group >= dev->ngroups || asid >= dev->nas)
+		return -EINVAL;
+
+	msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+	msg.req.vq_group_asid.group = group;
+	msg.req.vq_group_asid.asid = asid;
+
+	r = vduse_dev_msg_sync(dev, &msg);
+	if (r < 0)
+		return r;
+
+	vduse_set_group_asid_nomsg(dev, group, asid);
+	return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
 				struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
 	struct vduse_dev *dev = vdpa_to_vduse(vdpa);
 	int ret;
 
-	ret = vduse_domain_set_map(dev->domain, iotlb);
+	ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
 	if (ret)
 		return ret;
 
-	ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+	ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
 	if (ret) {
-		vduse_domain_clear_map(dev->domain, iotlb);
+		vduse_domain_clear_map(dev->as[asid].domain, iotlb);
 		return ret;
 	}
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
 	.get_vq_affinity	= vduse_vdpa_get_vq_affinity,
 	.reset			= vduse_vdpa_reset,
 	.set_map		= vduse_vdpa_set_map,
+	.set_group_asid		= vduse_set_group_asid,
 	.get_vq_map		= vduse_get_vq_map,
 	.free			= vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
 					     enum dma_data_direction dir)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
 
 	if (!token.group)
 		return;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
-
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
 	vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
 					     enum dma_data_direction dir)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
 
 	if (!token.group)
 		return;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
-
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
 	vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
 				     unsigned long attrs)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
+	dma_addr_t r;
 
 	if (!token.group)
 		return DMA_MAPPING_ERROR;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
+	r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);
 
-	return vduse_domain_map_page(domain, page, offset, size, dir, attrs);
+	return r;
 }
 
 static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
@@ -901,21 +969,25 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
 				 unsigned long attrs)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
 
 	if (!token.group)
 		return;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
-
-	return vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
+	vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
 }
 
 static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,
 				      dma_addr_t *dma_addr, gfp_t flag)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
 	unsigned long iova;
 	void *addr;
@@ -925,13 +997,14 @@ static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,
 		return NULL;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
 	addr = vduse_domain_alloc_coherent(domain, size,
 					   (dma_addr_t *)&iova, flag);
-	if (!addr)
-		return NULL;
-
-	*dma_addr = (dma_addr_t)iova;
+	if (addr)
+		*dma_addr = (dma_addr_t)iova;
 
 	return addr;
 }
@@ -941,29 +1014,39 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,
 				    unsigned long attrs)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
 
 	if (!token.group)
 		return;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
-
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
 	vduse_domain_free_coherent(domain, size, vaddr, dma_addr, attrs);
+	mutex_unlock(&vdev->domain_lock);
 }
 
 static bool vduse_dev_need_sync(union virtio_map token, dma_addr_t dma_addr)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
+	size_t bounce_size;
 
 	if (!token.group)
 		return false;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
+	bounce_size = domain->bounce_size;
 
-	return dma_addr < domain->bounce_size;
+	return dma_addr < bounce_size;
 }
 
 static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
@@ -976,15 +1059,22 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
 static size_t vduse_dev_max_mapping_size(union virtio_map token)
 {
 	struct vduse_dev *vdev;
+	struct vduse_as *as;
 	struct vduse_iova_domain *domain;
+	size_t bounce_size;
 
 	if (!token.group)
 		return 0;
 
 	vdev = token.group->dev;
-	domain = vdev->domain;
+	rcu_read_lock();
+	as = rcu_dereference(token.group->as);
+	domain = as->domain;
+	rcu_read_unlock();
+	domain = token.group->as->domain;
+	bounce_size = domain->bounce_size;
 
-	return domain->bounce_size;
+	return bounce_size;
 }
 
 static const struct virtio_map_ops vduse_map_ops = {
@@ -1124,39 +1214,40 @@ static int vduse_dev_queue_irq_work(struct vduse_dev *dev,
 	return ret;
 }
 
-static int vduse_dev_dereg_umem(struct vduse_dev *dev,
+static int vduse_dev_dereg_umem(struct vduse_dev *dev, u32 asid,
 				u64 iova, u64 size)
 {
 	int ret;
 
-	mutex_lock(&dev->mem_lock);
+	mutex_lock(&dev->as[asid].mem_lock);
 	ret = -ENOENT;
-	if (!dev->umem)
+	if (!dev->as[asid].umem)
 		goto unlock;
 
 	ret = -EINVAL;
-	if (!dev->domain)
+	if (!dev->as[asid].domain)
 		goto unlock;
 
-	if (dev->umem->iova != iova || size != dev->domain->bounce_size)
+	if (dev->as[asid].umem->iova != iova ||
+	    size != dev->as[asid].domain->bounce_size)
 		goto unlock;
 
-	vduse_domain_remove_user_bounce_pages(dev->domain);
-	unpin_user_pages_dirty_lock(dev->umem->pages,
-				    dev->umem->npages, true);
-	atomic64_sub(dev->umem->npages, &dev->umem->mm->pinned_vm);
-	mmdrop(dev->umem->mm);
-	vfree(dev->umem->pages);
-	kfree(dev->umem);
-	dev->umem = NULL;
+	vduse_domain_remove_user_bounce_pages(dev->as[asid].domain);
+	unpin_user_pages_dirty_lock(dev->as[asid].umem->pages,
+				    dev->as[asid].umem->npages, true);
+	atomic64_sub(dev->as[asid].umem->npages, &dev->as[asid].umem->mm->pinned_vm);
+	mmdrop(dev->as[asid].umem->mm);
+	vfree(dev->as[asid].umem->pages);
+	kfree(dev->as[asid].umem);
+	dev->as[asid].umem = NULL;
 	ret = 0;
 unlock:
-	mutex_unlock(&dev->mem_lock);
+	mutex_unlock(&dev->as[asid].mem_lock);
 	return ret;
 }
 
 static int vduse_dev_reg_umem(struct vduse_dev *dev,
-			      u64 iova, u64 uaddr, u64 size)
+			      u32 asid, u64 iova, u64 uaddr, u64 size)
 {
 	struct page **page_list = NULL;
 	struct vduse_umem *umem = NULL;
@@ -1164,14 +1255,14 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev,
 	unsigned long npages, lock_limit;
 	int ret;
 
-	if (!dev->domain || !dev->domain->bounce_map ||
-	    size != dev->domain->bounce_size ||
+	if (!dev->as[asid].domain || !dev->as[asid].domain->bounce_map ||
+	    size != dev->as[asid].domain->bounce_size ||
 	    iova != 0 || uaddr & ~PAGE_MASK)
 		return -EINVAL;
 
-	mutex_lock(&dev->mem_lock);
+	mutex_lock(&dev->as[asid].mem_lock);
 	ret = -EEXIST;
-	if (dev->umem)
+	if (dev->as[asid].umem)
 		goto unlock;
 
 	ret = -ENOMEM;
@@ -1195,7 +1286,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev,
 		goto out;
 	}
 
-	ret = vduse_domain_add_user_bounce_pages(dev->domain,
+	ret = vduse_domain_add_user_bounce_pages(dev->as[asid].domain,
 						 page_list, pinned);
 	if (ret)
 		goto out;
@@ -1208,7 +1299,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev,
 	umem->mm = current->mm;
 	mmgrab(current->mm);
 
-	dev->umem = umem;
+	dev->as[asid].umem = umem;
 out:
 	if (ret && pinned > 0)
 		unpin_user_pages(page_list, pinned);
@@ -1219,7 +1310,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev,
 		vfree(page_list);
 		kfree(umem);
 	}
-	mutex_unlock(&dev->mem_lock);
+	mutex_unlock(&dev->as[asid].mem_lock);
 	return ret;
 }
 
@@ -1251,47 +1342,66 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 
 	switch (cmd) {
 	case VDUSE_IOTLB_GET_FD: {
-		struct vduse_iotlb_entry entry;
+		struct vduse_iotlb_entry_v2 entry;
 		struct vhost_iotlb_map *map;
 		struct vdpa_map_file *map_file;
 		struct file *f = NULL;
+		u32 asid;
 
 		ret = -EFAULT;
-		if (copy_from_user(&entry, argp, sizeof(entry)))
-			break;
+		if (dev->api_version >= VDUSE_API_VERSION_1) {
+			if (copy_from_user(&entry, argp, sizeof(entry)))
+				break;
+		} else {
+			entry.asid = 0;
+			if (copy_from_user(&entry.v1, argp,
+					   sizeof(entry.v1)))
+				break;
+		}
 
 		ret = -EINVAL;
-		if (entry.start > entry.last)
+		if (entry.v1.start > entry.v1.last)
+			break;
+
+		if (entry.asid >= dev->nas)
 			break;
 
 		mutex_lock(&dev->domain_lock);
-		if (!dev->domain) {
+		asid = array_index_nospec(entry.asid, dev->nas);
+		if (!dev->as[asid].domain) {
 			mutex_unlock(&dev->domain_lock);
 			break;
 		}
-		spin_lock(&dev->domain->iotlb_lock);
-		map = vhost_iotlb_itree_first(dev->domain->iotlb,
-					      entry.start, entry.last);
+		spin_lock(&dev->as[asid].domain->iotlb_lock);
+		map = vhost_iotlb_itree_first(dev->as[asid].domain->iotlb,
+					      entry.v1.start, entry.v1.last);
 		if (map) {
 			map_file = (struct vdpa_map_file *)map->opaque;
 			f = get_file(map_file->file);
-			entry.offset = map_file->offset;
-			entry.start = map->start;
-			entry.last = map->last;
-			entry.perm = map->perm;
+			entry.v1.offset = map_file->offset;
+			entry.v1.start = map->start;
+			entry.v1.last = map->last;
+			entry.v1.perm = map->perm;
 		}
-		spin_unlock(&dev->domain->iotlb_lock);
+		spin_unlock(&dev->as[asid].domain->iotlb_lock);
 		mutex_unlock(&dev->domain_lock);
 		ret = -EINVAL;
 		if (!f)
 			break;
 
-		ret = -EFAULT;
-		if (copy_to_user(argp, &entry, sizeof(entry))) {
+		if (dev->api_version >= VDUSE_API_VERSION_1)
+			ret = copy_to_user(argp, &entry,
+					   sizeof(entry));
+		else
+			ret = copy_to_user(argp, &entry.v1,
+					   sizeof(entry.v1));
+
+		if (ret) {
+			ret = -EFAULT;
 			fput(f);
 			break;
 		}
-		ret = receive_fd(f, NULL, perm_to_file_flags(entry.perm));
+		ret = receive_fd(f, NULL, perm_to_file_flags(entry.v1.perm));
 		fput(f);
 		break;
 	}
@@ -1436,6 +1546,7 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 	}
 	case VDUSE_IOTLB_REG_UMEM: {
 		struct vduse_iova_umem umem;
+		u32 asid;
 
 		ret = -EFAULT;
 		if (copy_from_user(&umem, argp, sizeof(umem)))
@@ -1443,17 +1554,21 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 
 		ret = -EINVAL;
 		if (!is_mem_zero((const char *)umem.reserved,
-				 sizeof(umem.reserved)))
+				 sizeof(umem.reserved)) ||
+		    (dev->api_version < VDUSE_API_VERSION_1 &&
+		     umem.asid != 0) || umem.asid >= dev->nas)
 			break;
 
 		mutex_lock(&dev->domain_lock);
-		ret = vduse_dev_reg_umem(dev, umem.iova,
+		asid = array_index_nospec(umem.asid, dev->nas);
+		ret = vduse_dev_reg_umem(dev, asid, umem.iova,
 					 umem.uaddr, umem.size);
 		mutex_unlock(&dev->domain_lock);
 		break;
 	}
 	case VDUSE_IOTLB_DEREG_UMEM: {
 		struct vduse_iova_umem umem;
+		u32 asid;
 
 		ret = -EFAULT;
 		if (copy_from_user(&umem, argp, sizeof(umem)))
@@ -1461,10 +1576,15 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 
 		ret = -EINVAL;
 		if (!is_mem_zero((const char *)umem.reserved,
-				 sizeof(umem.reserved)))
+				 sizeof(umem.reserved)) ||
+		    (dev->api_version < VDUSE_API_VERSION_1 &&
+		     umem.asid != 0) ||
+		     umem.asid >= dev->nas)
 			break;
+
 		mutex_lock(&dev->domain_lock);
-		ret = vduse_dev_dereg_umem(dev, umem.iova,
+		asid = array_index_nospec(umem.asid, dev->nas);
+		ret = vduse_dev_dereg_umem(dev, asid, umem.iova,
 					   umem.size);
 		mutex_unlock(&dev->domain_lock);
 		break;
@@ -1472,6 +1592,7 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 	case VDUSE_IOTLB_GET_INFO: {
 		struct vduse_iova_info info;
 		struct vhost_iotlb_map *map;
+		u32 asid;
 
 		ret = -EFAULT;
 		if (copy_from_user(&info, argp, sizeof(info)))
@@ -1485,23 +1606,31 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 				 sizeof(info.reserved)))
 			break;
 
+		if (dev->api_version < VDUSE_API_VERSION_1) {
+			if (info.asid)
+				break;
+		} else if (info.asid >= dev->nas)
+			break;
+
 		mutex_lock(&dev->domain_lock);
-		if (!dev->domain) {
+		asid = array_index_nospec(info.asid, dev->nas);
+		if (!dev->as[asid].domain) {
 			mutex_unlock(&dev->domain_lock);
 			break;
 		}
-		spin_lock(&dev->domain->iotlb_lock);
-		map = vhost_iotlb_itree_first(dev->domain->iotlb,
+		spin_lock(&dev->as[asid].domain->iotlb_lock);
+		map = vhost_iotlb_itree_first(dev->as[asid].domain->iotlb,
 					      info.start, info.last);
 		if (map) {
 			info.start = map->start;
 			info.last = map->last;
 			info.capability = 0;
-			if (dev->domain->bounce_map && map->start == 0 &&
-			    map->last == dev->domain->bounce_size - 1)
+			if (dev->as[asid].domain->bounce_map &&
+			    map->start == 0 &&
+			    map->last == dev->as[asid].domain->bounce_size - 1)
 				info.capability |= VDUSE_IOVA_CAP_UMEM;
 		}
-		spin_unlock(&dev->domain->iotlb_lock);
+		spin_unlock(&dev->as[asid].domain->iotlb_lock);
 		mutex_unlock(&dev->domain_lock);
 		if (!map)
 			break;
@@ -1526,8 +1655,10 @@ static int vduse_dev_release(struct inode *inode, struct file *file)
 	struct vduse_dev *dev = file->private_data;
 
 	mutex_lock(&dev->domain_lock);
-	if (dev->domain)
-		vduse_dev_dereg_umem(dev, 0, dev->domain->bounce_size);
+	for (int i = 0; i < dev->nas; i++)
+		if (dev->as[i].domain)
+			vduse_dev_dereg_umem(dev, i, 0,
+					     dev->as[i].domain->bounce_size);
 	mutex_unlock(&dev->domain_lock);
 	spin_lock(&dev->msg_lock);
 	/* Make sure the inflight messages can processed after reconncection */
@@ -1746,7 +1877,6 @@ static struct vduse_dev *vduse_dev_create(void)
 		return NULL;
 
 	mutex_init(&dev->lock);
-	mutex_init(&dev->mem_lock);
 	mutex_init(&dev->domain_lock);
 	spin_lock_init(&dev->msg_lock);
 	INIT_LIST_HEAD(&dev->send_list);
@@ -1797,8 +1927,11 @@ static int vduse_destroy_dev(char *name)
 	idr_remove(&vduse_idr, dev->minor);
 	kvfree(dev->config);
 	vduse_dev_deinit_vqs(dev);
-	if (dev->domain)
-		vduse_domain_destroy(dev->domain);
+	for (int i = 0; i < dev->nas; i++) {
+		if (dev->as[i].domain)
+			vduse_domain_destroy(dev->as[i].domain);
+	}
+	kfree(dev->as);
 	kfree(dev->name);
 	kfree(dev->groups);
 	vduse_dev_destroy(dev);
@@ -1845,12 +1978,17 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
 			 sizeof(config->reserved)))
 		return false;
 
-	if (api_version < VDUSE_API_VERSION_1 && config->ngroups)
+	if (api_version < VDUSE_API_VERSION_1 &&
+	    (config->ngroups || config->nas))
 		return false;
 
-	if (api_version >= VDUSE_API_VERSION_1 &&
-	    (!config->ngroups || config->ngroups > VDUSE_DEV_MAX_GROUPS))
-		return false;
+	if (api_version >= VDUSE_API_VERSION_1) {
+		if (!config->ngroups || config->ngroups > VDUSE_DEV_MAX_GROUPS)
+			return false;
+
+		if (!config->nas || config->nas > VDUSE_DEV_MAX_AS)
+			return false;
+	}
 
 	if (config->vq_align > PAGE_SIZE)
 		return false;
@@ -1915,7 +2053,8 @@ static ssize_t bounce_size_store(struct device *device,
 
 	ret = -EPERM;
 	mutex_lock(&dev->domain_lock);
-	if (dev->domain)
+	/* Assuming that if the first domain is allocated, all are allocated */
+	if (dev->as[0].domain)
 		goto unlock;
 
 	ret = kstrtouint(buf, 10, &bounce_size);
@@ -1977,6 +2116,13 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	for (u32 i = 0; i < dev->ngroups; ++i)
 		dev->groups[i].dev = dev;
 
+	dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
+	dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
+	if (!dev->as)
+		goto err_as;
+	for (int i = 0; i < dev->nas; i++)
+		mutex_init(&dev->as[i].mem_lock);
+
 	dev->name = kstrdup(config->name, GFP_KERNEL);
 	if (!dev->name)
 		goto err_str;
@@ -2013,6 +2159,8 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 err_idr:
 	kfree(dev->name);
 err_str:
+	kfree(dev->as);
+err_as:
 	kfree(dev->groups);
 err_vq_groups:
 	vduse_dev_destroy(dev);
@@ -2138,7 +2286,7 @@ static int vduse_dev_init_vdpa(struct vduse_dev *dev, const char *name)
 
 	vdev = vdpa_alloc_device(struct vduse_vdpa, vdpa, dev->dev,
 				 &vduse_vdpa_config_ops, &vduse_map_ops,
-				 dev->ngroups, 1, name, true);
+				 dev->ngroups, dev->nas, name, true);
 	if (IS_ERR(vdev))
 		return PTR_ERR(vdev);
 
@@ -2153,7 +2301,8 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 			const struct vdpa_dev_set_config *config)
 {
 	struct vduse_dev *dev;
-	int ret;
+	size_t domain_bounce_size;
+	int ret, i;
 
 	mutex_lock(&vduse_lock);
 	dev = vduse_find_dev(name);
@@ -2167,29 +2316,38 @@ static int vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 		return ret;
 
 	mutex_lock(&dev->domain_lock);
-	if (!dev->domain)
-		dev->domain = vduse_domain_create(VDUSE_IOVA_SIZE - 1,
-						  dev->bounce_size);
-	mutex_unlock(&dev->domain_lock);
-	if (!dev->domain) {
-		ret = -ENOMEM;
-		goto domain_err;
+	ret = 0;
+
+	domain_bounce_size = dev->bounce_size / dev->nas;
+	for (i = 0; i < dev->nas; ++i) {
+		dev->as[i].domain = vduse_domain_create(VDUSE_IOVA_SIZE - 1,
+							domain_bounce_size);
+		if (!dev->as[i].domain) {
+			ret = -ENOMEM;
+			goto err;
+		}
 	}
 
+	mutex_unlock(&dev->domain_lock);
+
 	ret = _vdpa_register_device(&dev->vdev->vdpa, dev->vq_num);
-	if (ret) {
-		goto register_err;
-	}
+	if (ret)
+		goto err_register;
 
 	return 0;
 
-register_err:
+err_register:
 	mutex_lock(&dev->domain_lock);
-	vduse_domain_destroy(dev->domain);
-	dev->domain = NULL;
+
+err:
+	for (int j = 0; j < i; j++) {
+		if (dev->as[j].domain) {
+			vduse_domain_destroy(dev->as[j].domain);
+			dev->as[j].domain = NULL;
+		}
+	}
 	mutex_unlock(&dev->domain_lock);
 
-domain_err:
 	put_device(&dev->vdev->vdpa.dev);
 
 	return ret;
include/uapi/linux/vduse.h +50 -3

119 lines not on this page · raw message

--
2.51.1

#6 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

More below.

Quoted 150 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 85 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
                return;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 30 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7 -2
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                     unsigned long attrs)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
+       dma_addr_t r;
 

drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5 -1
        if (!token.group)
                return DMA_MAPPING_ERROR;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
+       r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);

Same here.

Thanks

author maintainer
#7 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

Quoted 87 lines
Quoted 87 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

The function vduse_domain uses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.

Quoted 45 lines
Quoted 30 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7 -2
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                     unsigned long attrs)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
+       dma_addr_t r;
 

drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5 -1
        if (!token.group)
                return DMA_MAPPING_ERROR;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
+       r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);

Same here.

Thanks

#8 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

1) should we need to wait for the DMA to be completed before assigning to the new as
2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 4 lines

The function vduse_domain uses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.

Thanks

Quoted 47 lines
Quoted 46 lines
Quoted 31 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7 -2
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                     unsigned long attrs)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
+       dma_addr_t r;
 

drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5 -1
        if (!token.group)
                return DMA_MAPPING_ERROR;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
+       r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);

Same here.

Thanks

author maintainer
#9 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent. I'm ok if it takes the one before the concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

Quoted 7 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

Quoted 3 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

No, the group will outlive the operation as it is bound to the device.

Quoted 55 lines
Quoted 4 lines

The function vduse_domain uses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.

Thanks

Quoted 47 lines
Quoted 46 lines
Quoted 31 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7 -2
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                     unsigned long attrs)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
+       dma_addr_t r;
 

drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5 -1
        if (!token.group)
                return DMA_MAPPING_ERROR;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
+       r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);

Same here.

Thanks

#10 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 341 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 3 lines

}
read_unlock(&token.group->as->domain);
}

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done. But this is not the logic that is implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg(). We need to explain why set_group_asid() doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 6 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc. But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

Anyhow, tracking and failing seems to be more robust. For example, kernel swiotlb has a layer to track whether the address that is being unmapped is mapped before. In our case we should make sure when race happens, isolation via ASID won't be broken.

Thanks

Quoted 57 lines
Quoted 56 lines
Quoted 5 lines

The function vduse_domain uses its own lock to protect concurrent access to the maps of the ASID itself, as they were needed before implementing ASID already.

Thanks

Quoted 47 lines
Quoted 46 lines
Quoted 31 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7 -2
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                     unsigned long attrs)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
+       dma_addr_t r;
 

drivers/vdpa/vdpa_user/vduse_dev.c static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, · line 962

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5 -1
        if (!token.group)
                return DMA_MAPPING_ERROR;
 
        vdev = token.group->dev;
-       domain = vdev->domain;
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
+       r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);

Same here.

Thanks

author maintainer
#11 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 348 lines · Jason Wang
Quoted 341 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 3 lines

}
read_unlock(&token.group->as->domain);
}

Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

Quoted 3 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Quoted 4 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 40 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

Quoted 8 lines

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 4 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID? It seems hard to communicate that the ASID has changed to the DMA operation callback.

Quoted 5 lines

For example,
kernel swiotlb has a layer to track whether the address that is being unmapped is mapped before. In our case we should make sure when race happens, isolation via ASID won't be broken.

Right but it cannot protect if the iotlb is valid in the second ASID but just points to a different physical / virtual page.

#12 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 7 lines · Eugenio Perez Martin
Quoted 3 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()?

synchronize_rcu is called after writer makes it's changes.

Quoted 2 lines

What do we need to do on the other side of
syncrhonize_rcu()?

Presumably, return so the caller knows the as has been updated.

However, user-triggerable synchronize_rcu() is almost always a bug.

If that's what is going on, you want srcu.

Signature · 1 line
MST
author maintainer
#13 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Eugenio Perez Martin
Quoted 3 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()?

Quoted 7 lines

synchronize_rcu is called after writer makes it's changes.

Quoted 2 lines

What do we need to do on the other side of
syncrhonize_rcu()?

Presumably, return so the caller knows the as has been updated.

I'm happy to add the syncrhonize_rcu() just in case, but the caller of vduse_set_group_asid_nomsg does not need to know that the reader has been updated.

The first caller is vduse_dev_reset which has its own way to avoid being called while vqs are being processed. In particular, it reset their addresses which is way more dangerous. I could call it with dev->rwsem down though.

The second one is set_group_asid vdpa callback which is called from the ioctl itself.

Moreover, rcu_assign_pointer is WRITE_ONCE by itself so we know all the readers will get the new value after it. So what's the value of explicitly waiting for all the readers to finalize their DMA operation?

I'd understand if we need to do modifications or memory management in the now unused ASID, but that's not the case here.

Quoted 4 lines

However, user-triggerable synchronize_rcu() is almost always a bug.

If that's what is going on, you want srcu.

I did not know about it, thanks! but I think all the code that can sleep is out of the RCU critical sections now, isn't it?

#14 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 23 lines · Eugenio Perez Martin
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Eugenio Perez Martin
Quoted 3 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()?

Quoted 7 lines

synchronize_rcu is called after writer makes it's changes.

Quoted 2 lines

What do we need to do on the other side of
syncrhonize_rcu()?

Presumably, return so the caller knows the as has been updated.

I'm happy to add the syncrhonize_rcu() just in case, but the caller of vduse_set_group_asid_nomsg does not need to know that the reader has been updated.

Quoted 7 lines

The first caller is vduse_dev_reset which has its own way to avoid being called while vqs are being processed. In particular, it reset their addresses which is way more dangerous. I could call it with dev->rwsem down though.

The second one is set_group_asid vdpa callback which is called from the ioctl itself.

This one for sure needs to know that after ioctl completed the new AS is in use.

Quoted 2 lines

Moreover, rcu_assign_pointer is WRITE_ONCE by itself so we know all the readers will get the new value after it.

No we don't. The words "after it" do not have a meaning on an SMP system. The only way to know that another CPU sees a value is to synchronize with it.

Quoted 2 lines

So what's the value of
explicitly waiting for all the readers to finalize their DMA operation?

Read Documentation/memory-barriers.txt
If still unclear, read it again ))

Quoted 10 lines

I'd understand if we need to do modifications or memory management in the now unused ASID, but that's not the case here.

Quoted 4 lines

However, user-triggerable synchronize_rcu() is almost always a bug.

If that's what is going on, you want srcu.

I did not know about it, thanks! but I think all the code that can sleep is out of the RCU critical sections now, isn't it?

what I tried to say is that if triggering synchronize_rcu from userspace is a problem, srcu is one solution.

#15 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

In reply to Eugenio Perez Martin
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 34 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

Quoted 4 lines

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?

Quoted 13 lines
Quoted 5 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 5 lines
Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?

If possible, it would be better.

Quoted 2 lines

It seems hard to
communicate that the ASID has changed to the DMA operation callback.

Maybe we can encode this into iova.

Quoted 10 lines
Quoted 6 lines

For example,
kernel swiotlb has a layer to track whether the address that is being unmapped is mapped before. In our case we should make sure when race happens, isolation via ASID won't be broken.

Right but it cannot protect if the iotlb is valid in the second ASID but just points to a different physical / virtual page.

Thanks

author maintainer
#16 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 34 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

Quoted 4 lines

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 8 lines

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?

Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.

There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.

From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.

Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:

void* thread0_func(void* arg) {

struct {
        int vq_group = 0,
        int iova, size, perm, ...
    } s;
    int fd = (intptr_t)arg;

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
    // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
    // TODO: Wait until thread0 complete SET_GROUP_ASID

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);

    return NULL;
}
void* thread1_func(void* arg) {
    struct vhost_vring_state s = {
        .index = 0,
        .num = 1,
    };
    int fd = (int)(intptr_t)arg;

    // TODO: Wait until thread2 calls dma_map
    ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
    // TODO: Signal thread2 that can proceed with dma_unmap

    return NULL;
}
int main() {
    pthread_t thread0, thread1;
    int fd = open("/dev/vhost-vdpa-0", ...)

    pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd);
    pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    return EXIT_SUCCESS;
}

---

We need something to synchronize at userland level here, filling the TODOs.

We can replace the hypothetical VHOST_VDPA_VQ_GROUP_DMA_MAP and _UNMAP with access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.

We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So DMA_MAP and DMA_UNMAP does not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.

Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?

If possible, it would be better.

It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.

Quoted 5 lines
Quoted 2 lines

It seems hard to
communicate that the ASID has changed to the DMA operation callback.

Maybe we can encode this into iova.

I'm adding the full rwlock, but can you expand on your idea on this? It would be great to have it documented in case we need future optimizations.

#17 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 528 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 34 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

Quoted 4 lines

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 8 lines

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?

Quoted 8 lines

Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.

There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.

Probably.

Quoted 27 lines

From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.

Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:

void* thread0_func(void* arg) {

struct {
        int vq_group = 0,
        int iova, size, perm, ...
    } s;
    int fd = (intptr_t)arg;

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
    // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
    // TODO: Wait until thread0 complete SET_GROUP_ASID

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);

    return NULL;
}
Quoted 8 lines
void* thread1_func(void* arg) {
    struct vhost_vring_state s = {
        .index = 0,
        .num = 1,
    };
    int fd = (int)(intptr_t)arg;

    // TODO: Wait until thread2 calls dma_map

This is something exactly rwlock or synchronize_rcu() can do? Or is this the charge of the vDPA parent to do that?

If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via mmap() or umem_reg.

Looking at existing implementation:

- for mlx5e, it looks like it assumes the set_group_asid() works only without DRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated

Quoted 2 lines

ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed with dma_unmap

But the issue I mention is that the, from the view of the vDPA bus:

1) it offers set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is used

So theoretically, set_group_as_id() could happen between

dma_addr = dma_map();

and

dma_unmap(dma_adrr);

But those two dma_addr refers to the different address space. Instead of trying to do synchronization, maybe we can simply fail set_group_asid if DRIVER_OK is set.

Quoted 26 lines
    return NULL;
}
int main() {
    pthread_t thread0, thread1;
    int fd = open("/dev/vhost-vdpa-0", ...)

    pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd);
    pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    return EXIT_SUCCESS;
}

---

We need something to synchronize at userland level here, filling the TODOs.

We can replace the hypothetical VHOST_VDPA_VQ_GROUP_DMA_MAP and _UNMAP with access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.

Quoted 5 lines

We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So DMA_MAP and DMA_UNMAP does not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.

See above.

Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?

If possible, it would be better.

It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

It seems hard to
communicate that the ASID has changed to the DMA operation callback.

Maybe we can encode this into iova.

I'm adding the full rwlock, but can you expand on your idea on this?

Encode the asid to the upper bits of IOVA, so when doing dma_unmap() we can compare the group/as with the one that is encoded in dma_addr. If it differs, warn or bug.

If we can make sure there's no set_group_asid() when DRIVER_OK is set, it's not a must then (or could be treated as a kind of hardening).

Quoted 3 lines

It would be great to have it documented in case we need future optimizations.

Thanks

author maintainer
#18 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 575 lines · Jason Wang
Quoted 528 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 34 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

Quoted 4 lines

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 8 lines

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?

Quoted 8 lines

Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.

There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.

Probably.

Quoted 34 lines

From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.

Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:

void* thread0_func(void* arg) {

struct {
        int vq_group = 0,
        int iova, size, perm, ...
    } s;
    int fd = (intptr_t)arg;

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
    // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
    // TODO: Wait until thread0 complete SET_GROUP_ASID

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);

    return NULL;
}
void* thread1_func(void* arg) {
    struct vhost_vring_state s = {
        .index = 0,
        .num = 1,
    };
    int fd = (int)(intptr_t)arg;
Quoted 5 lines
Quoted 1 line
    // TODO: Wait until thread2 calls dma_map

This is something exactly rwlock or synchronize_rcu() can do? Or is this the charge of the vDPA parent to do that?

No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.

If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:

cpu0] DMA map ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Now we have two possibilities: either cpu0] DMA_UNMAP is called or cpu1] set_group_asid is called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:

cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock

If set_group_asid run first by chance, ASID 1 is the one that is updated:
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock
cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the set_map call. I'm not adding the synchronize_rcu call because we stated it should not be called from userland, but the outcome is similar.

Let me put another example: It's like calling fdup() and write() from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of the writes() is just not predictable. And the kernel just cannot make it predictable.

static int fd_a;
static int fd_b;

void *write_thread(void *arg) {
    const char *msg = "Writing to fd_b\n";
    write(fd_b, msg, strlen(msg));
    return NULL;
}
void *dup_thread(void *arg) {
    dup2(fd_a, fd_b);
    return NULL;
}
int main() {
    pthread_t writer, duper;

    fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644);

    pthread_create(&writer, NULL, write_thread, NULL);
    pthread_create(&duper, NULL, dup_thread, NULL);

    pthread_join(writer, NULL);
    pthread_join(duper, NULL);

    close(fd_a);
    close(fd_b);

    return 0;
}

--

Quoted 11 lines

If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via mmap() or umem_reg.

Looking at existing implementation:

- for mlx5e, it looks like it assumes the set_group_asid() works only without DRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated

Yes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.

Quoted 10 lines
Quoted 2 lines

ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed with dma_unmap

But the issue I mention is that the, from the view of the vDPA bus:

1) it offers set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is used

So theoretically, set_group_as_id() could happen between

Quoted 7 lines

dma_addr = dma_map();

and

dma_unmap(dma_adrr);

But those two dma_addr refers to the different address space.

I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?

Quoted 4 lines

Instead
of trying to do synchronization, maybe we can simply fail set_group_asid if DRIVER_OK is set.

That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.

Quoted 64 lines
Quoted 26 lines
    return NULL;
}
int main() {
    pthread_t thread0, thread1;
    int fd = open("/dev/vhost-vdpa-0", ...)

    pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd);
    pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    return EXIT_SUCCESS;
}

---

We need something to synchronize at userland level here, filling the TODOs.

We can replace the hypothetical VHOST_VDPA_VQ_GROUP_DMA_MAP and _UNMAP with access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.

Quoted 5 lines

We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So DMA_MAP and DMA_UNMAP does not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.

See above.

Quoted 30 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?

If possible, it would be better.

It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.

Quoted 3 lines
Quoted 2 lines

It seems hard to
communicate that the ASID has changed to the DMA operation callback.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Maybe we can encode this into iova.

I'm adding the full rwlock, but can you expand on your idea on this?

Encode the asid to the upper bits of IOVA, so when doing dma_unmap() we can compare the group/as with the one that is encoded in dma_addr. If it differs, warn or bug.

But the ASID is already a parameter in the dma_unmap. I thought you meant to encode in the device's memory read.

Quoted 9 lines

If we can make sure there's no set_group_asid() when DRIVER_OK is set, it's not a must then (or could be treated as a kind of hardening).

Quoted 3 lines

It would be great to have it documented in case we need future optimizations.

Thanks

#19 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 554 lines · Eugenio Perez Martin
Quoted 552 lines · Jason Wang
Quoted 528 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 34 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

Quoted 4 lines

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 8 lines

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?

Quoted 8 lines

Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.

There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.

Probably.

Quoted 11 lines

From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.

Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

void* thread0_func(void* arg) {

struct {
        int vq_group = 0,
        int iova, size, perm, ...
    } s;
    int fd = (intptr_t)arg;

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);

I don't get the semantic of VHOST_VDPA_VQ_GROUP_DMA_MAP. And it's probably wrong to allow userspace to map based on group.

We have VHOST_IOTLB_UPDATE which map pages based on as (asid) this is correct.

More below

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
    // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
    // TODO: Wait until thread0 complete SET_GROUP_ASID

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);

    return NULL;
}
void* thread1_func(void* arg) {
    struct vhost_vring_state s = {
        .index = 0,
        .num = 1,
    };
    int fd = (int)(intptr_t)arg;

    // TODO: Wait until thread2 calls dma_map
Quoted 7 lines
Quoted 3 lines

This is something exactly rwlock or synchronize_rcu() can do? Or is this the charge of the vDPA parent to do that?

No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.

Ok, basically I think we are talking about different things. That's fine.

If I understand you correctly, you mean we need to synchronize between IOTLB updating (IOTLB_UPDATE/IOTLB_INVALIDATE) and set_group_asid()?

This sounds unnecessary since:

1) IOTLB_UPDATE/IOTLB_INVALIDATE is updating the address space internal mappings
2) set_group_asid() is to assign an AS to a group

Quoted 35 lines

If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:

cpu0] DMA map ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Now we have two possibilities: either cpu0] DMA_UNMAP is called or cpu1] set_group_asid is called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:

cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock

If set_group_asid run first by chance, ASID 1 is the one that is updated:
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock
cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Quoted 7 lines

On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the set_map call. I'm not adding the synchronize_rcu call because we stated it should not be called from userland, but the outcome is similar.

I think we should figure out if VHOST_VDPA_VQ_GROUP_DMA_MAP is useful or not.

Quoted 72 lines

Let me put another example: It's like calling fdup() and write() from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of the writes() is just not predictable. And the kernel just cannot make it predictable.

static int fd_a;
static int fd_b;

void *write_thread(void *arg) {
    const char *msg = "Writing to fd_b\n";
    write(fd_b, msg, strlen(msg));
    return NULL;
}
void *dup_thread(void *arg) {
    dup2(fd_a, fd_b);
    return NULL;
}
int main() {
    pthread_t writer, duper;

    fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644);

    pthread_create(&writer, NULL, write_thread, NULL);
    pthread_create(&duper, NULL, dup_thread, NULL);

    pthread_join(writer, NULL);
    pthread_join(duper, NULL);

    close(fd_a);
    close(fd_b);

    return 0;
}

--

Quoted 11 lines

If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via mmap() or umem_reg.

Looking at existing implementation:

- for mlx5e, it looks like it assumes the set_group_asid() works only without DRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated

Yes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.

Quoted 16 lines
Quoted 2 lines

ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed with dma_unmap

But the issue I mention is that the, from the view of the vDPA bus:

1) it offers set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is used

So theoretically, set_group_as_id() could happen between

dma_addr = dma_map();

and

dma_unmap(dma_adrr);

Quoted 7 lines
Quoted 1 line

But those two dma_addr refers to the different address space.

I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?

Just to clarify, I for dma_map()/dma_unmap() this is not the UAPI part (and we don't have that). I basically mean the dma API which is used by virtio-vDPA.

Quoted 10 lines
Quoted 5 lines

Instead
of trying to do synchronization, maybe we can simply fail set_group_asid if DRIVER_OK is set.

That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.

Did you mean the SVQ may update the group asid while DRIVER_OK is set? If yes, we need to fix that.

Quoted 70 lines
Quoted 70 lines
Quoted 27 lines

    return NULL;
}
int main() {
    pthread_t thread0, thread1;
    int fd = open("/dev/vhost-vdpa-0", ...)

    pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd);
    pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    return EXIT_SUCCESS;
}

---

We need something to synchronize at userland level here, filling the TODOs.

We can replace the hypothetical VHOST_VDPA_VQ_GROUP_DMA_MAP and _UNMAP with access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.

Quoted 5 lines

We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So DMA_MAP and DMA_UNMAP does not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.

See above.

Quoted 34 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?

If possible, it would be better.

It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.

Quoted 5 lines
Quoted 2 lines

It seems hard to
communicate that the ASID has changed to the DMA operation callback.

Maybe we can encode this into iova.

I'm adding the full rwlock, but can you expand on your idea on this?

Quoted 7 lines
Quoted 4 lines

Encode the asid to the upper bits of IOVA, so when doing dma_unmap() we can compare the group/as with the one that is encoded in dma_addr. If it differs, warn or bug.

But the ASID is already a parameter in the dma_unmap. I thought you meant to encode in the device's memory read.

To avoid unmap a address that is not belong to the this ASID.

Quoted 10 lines
Quoted 10 lines

If we can make sure there's no set_group_asid() when DRIVER_OK is set, it's not a must then (or could be treated as a kind of hardening).

Quoted 3 lines

It would be great to have it documented in case we need future optimizations.

Thanks

Thanks

author maintainer
#20 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 525 lines · Jason Wang
Quoted 523 lines · Eugenio Perez Martin
Quoted 521 lines · Jason Wang
Quoted 519 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +20 -6
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
        struct mm_struct *mm;
 };
 
+struct vduse_as {
+       struct vduse_iova_domain *domain;
+       struct vduse_umem *umem;
+       struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+       struct vduse_as *as __rcu;
        struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
        struct vduse_vdpa *vdev;
        struct device *dev;
        struct vduse_virtqueue **vqs;
-       struct vduse_iova_domain *domain;
+       struct vduse_as *as;
        char *name;
        struct mutex lock;
        spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
        u32 vq_num;
        u32 vq_align;
        u32 ngroups;
-       struct vduse_umem *umem;
+       u32 nas;
        struct vduse_vq_group *groups;
-       struct mutex mem_lock;
        unsigned int bounce_size;
        struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
        return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                  u64 start, u64 last)
 {
        struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                return -EINVAL;
 
        msg.req.type = VDUSE_UPDATE_IOTLB;
-       msg.req.iova.start = start;
-       msg.req.iova.last = last;
+       if (dev->api_version < VDUSE_API_VERSION_1) {
+               msg.req.iova.start = start;
+               msg.req.iova.last = last;
+       } else {
+               msg.req.iova_v2.start = start;
+               msg.req.iova_v2.last = last;
+               msg.req.iova_v2.asid = asid;
+       }
 
        return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
        return mask;
 }
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static __poll_t vduse_dev_poll(struct file *file, poll_table *wait) · line 462

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                      unsigned int group, unsigned int asid)
+{
+       /*
+        * Two concurrent updates to this pointer are valid as they cannot
+        * point to an invalid region.  It is ok for them to race as long as
+        * the readers see a consistent state through RCU.
+        */
+       rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines
drivers/vdpa/vdpa_user/vduse_dev.c +39 -6
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
        int i;
-       struct vduse_iova_domain *domain = dev->domain;
 
        /* The coherent mappings are handled in vduse_dev_free_coherent() */
-       if (domain && domain->bounce_map)
-               vduse_domain_reset_bounce_map(domain);
+       for (i = 0; i < dev->nas; i++) {
+               struct vduse_iova_domain *domain = dev->as[i].domain;
+
+               if (domain && domain->bounce_map)
+                       vduse_domain_reset_bounce_map(domain);
+       }
+
+       for (i = 0; i < dev->ngroups; i++)
+               vduse_set_group_asid_nomsg(dev, i, 0);
 
        down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
        return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                               unsigned int asid)
+{
+       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+       struct vduse_dev_msg msg = { 0 };
+       int r;
+
+       if (dev->api_version < VDUSE_API_VERSION_1 ||
+           group >= dev->ngroups || asid >= dev->nas)
+               return -EINVAL;
+
+       msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+       msg.req.vq_group_asid.group = group;
+       msg.req.vq_group_asid.asid = asid;
+
+       r = vduse_dev_msg_sync(dev, &msg);
+       if (r < 0)
+               return r;
+
+       vduse_set_group_asid_nomsg(dev, group, asid);
+       return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret;
 
-       ret = vduse_domain_set_map(dev->domain, iotlb);
+       ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
        if (ret)
                return ret;
 
-       ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+       ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
        if (ret) {
-               vduse_domain_clear_map(dev->domain, iotlb);
+               vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                return ret;
        }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
        .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
        .reset                  = vduse_vdpa_reset,
        .set_map                = vduse_vdpa_set_map,
+       .set_group_asid         = vduse_set_group_asid,
        .get_vq_map             = vduse_get_vq_map,
        .free                   = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                             enum dma_data_direction dir)
 {
        struct vduse_dev *vdev;
+       struct vduse_as *as;
        struct vduse_iova_domain *domain;
 
        if (!token.group)
                return;
 
Quoted 13 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_dev_sync_single_for_device(union virtio_map token, · line 922

Quoted 8 lines
drivers/vdpa/vdpa_user/vduse_dev.c +4 -2
        vdev = token.group->dev;
-       domain = vdev->domain;
-
+       rcu_read_lock();
+       as = rcu_dereference(token.group->as);
+       domain = as->domain;
+       rcu_read_unlock();
        vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 38 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

[1]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

cpu0 may read as1 while it wants as0 actually?

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.

There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.

[2]

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Probably.

Quoted 5 lines

From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.

[3]

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:

void* thread0_func(void* arg) {
struct {
int vq_group = 0,
int iova, size, perm, ...
} s;
int fd = (intptr_t)arg;

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);

I don't get the semantic of VHOST_VDPA_VQ_GROUP_DMA_MAP. And it's probably wrong to allow userspace to map based on group.

We have VHOST_IOTLB_UPDATE which map pages based on as (asid) this is correct.

Yes, I said that at [2], in reply to your scenario at [1] :). Did I misunderstand?

Quoted 29 lines

More below

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
    // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
    // TODO: Wait until thread0 complete SET_GROUP_ASID

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);

    return NULL;
}
void* thread1_func(void* arg) {
    struct vhost_vring_state s = {
        .index = 0,
        .num = 1,
    };
    int fd = (int)(intptr_t)arg;

    // TODO: Wait until thread2 calls dma_map
Quoted 7 lines
Quoted 3 lines

This is something exactly rwlock or synchronize_rcu() can do? Or is this the charge of the vDPA parent to do that?

No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.

Ok, basically I think we are talking about different things. That's fine.

Quoted 8 lines

If I understand you correctly, you mean we need to synchronize between IOTLB updating (IOTLB_UPDATE/IOTLB_INVALIDATE) and set_group_asid()?

This sounds unnecessary since:

1) IOTLB_UPDATE/IOTLB_INVALIDATE is updating the address space internal mappings
2) set_group_asid() is to assign an AS to a group

Right, we're on the same page here [3].

Quoted 43 lines
Quoted 35 lines

If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:

cpu0] DMA map ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Now we have two possibilities: either cpu0] DMA_UNMAP is called or cpu1] set_group_asid is called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:

cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock

If set_group_asid run first by chance, ASID 1 is the one that is updated:
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock
cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Quoted 7 lines

On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the set_map call. I'm not adding the synchronize_rcu call because we stated it should not be called from userland, but the outcome is similar.

Quoted 2 lines

I think we should figure out if VHOST_VDPA_VQ_GROUP_DMA_MAP is useful or not.

It's not. I just wanted to expand how I understood your scenario, but let me know if I missed something!

Quoted 80 lines
Quoted 72 lines

Let me put another example: It's like calling fdup() and write() from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of the writes() is just not predictable. And the kernel just cannot make it predictable.

static int fd_a;
static int fd_b;

void *write_thread(void *arg) {
    const char *msg = "Writing to fd_b\n";
    write(fd_b, msg, strlen(msg));
    return NULL;
}
void *dup_thread(void *arg) {
    dup2(fd_a, fd_b);
    return NULL;
}
int main() {
    pthread_t writer, duper;

    fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644);

    pthread_create(&writer, NULL, write_thread, NULL);
    pthread_create(&duper, NULL, dup_thread, NULL);

    pthread_join(writer, NULL);
    pthread_join(duper, NULL);

    close(fd_a);
    close(fd_b);

    return 0;
}

--

Quoted 11 lines

If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via mmap() or umem_reg.

Looking at existing implementation:

- for mlx5e, it looks like it assumes the set_group_asid() works only without DRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated

Yes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.

Quoted 16 lines
Quoted 2 lines

ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed with dma_unmap

But the issue I mention is that the, from the view of the vDPA bus:

1) it offers set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is used

So theoretically, set_group_as_id() could happen between

dma_addr = dma_map();

and

dma_unmap(dma_adrr);

Quoted 7 lines
Quoted 1 line

But those two dma_addr refers to the different address space.

I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?

Quoted 4 lines

Just to clarify, I for dma_map()/dma_unmap() this is not the UAPI part (and we don't have that). I basically mean the dma API which is used by virtio-vDPA.

So let's say that the virtio_vdpa driver is able to call .set_group_asid just to explain how much it is protected:

->vduse_dev_map_page and ->vduse_dev_unmap_page
In this series, rcu_dereference makes sure that only the stack copy of the domain pointer is used, but the next version the code will have a read_lock taken protecting that set_group_asid actions on token.group->as cannot run at the same time that the vduse_domain_map_page(...) or vduse_domain_unmap_page(...) function call run in cpu0:

static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs)

{
        struct vduse_iova_domain *domain;
        dma_addr_t r;

        if (!token.group)
                return DMA_MAPPING_ERROR;

        read_lock(&dev->groups[group].as_lock);
        domain = token.group->as->domain;
        r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);
        read_unlock(&dev->groups[group].as_lock);

        return r;
}

static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
size_t size, enum dma_data_direction dir, unsigned long attrs)

{
        struct vduse_iova_domain *domain;

        if (!token.group)
                return;

        read_lock(&dev->groups[group].as_lock);
        domain = token.group->as->domain;
        vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
        read_unlock(&dev->groups[group].as_lock);
}

---

-> vduse_set_group_asid will have the corresponding write_lock, so it cannot modify any vq group AS pointer while they're mapping or unmapping pages from a domain:
static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
unsigned int group, unsigned int asid)

{
        write_lock(&dev->groups[group].as_lock);
        dev->groups[group].as = &dev->as[asid];
        write_unlock(&dev->groups[group].as_lock);
}

---

Now your question:

assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?
--

Yes, it can happen even with the rwlock or any other synchronization in VDUSE, as it does not order the vdpa core calls.

This is the code I have in mind to prove this, let me know if I'm missing something from your scenario:

static int thread1_func(void *data)

{
    vduse_dev_map_page(token, ...);
    printk(KERN_INFO "map_page called\n");

    // TODO: Need a way to know if the group ASID has changed to
    // know if this unmap is valid or not.
    vduse_dev_unmap_page(token, ...);
    printk(KERN_INFO "unmap_page called\n");

    return 0;
}

static int thread2_func(void *data)

{
    u32 vq_group = 0;
    u32 asid = 1;

    vdpa->ops->set_group_asid(global_vdpa, vq_group, asid);
    printk(KERN_INFO "set_group_asid called\n");

    return 0;
}

static int __init kernel_threads_init(void)

{
    struct task_struct *thread1, *thread2;

    thread1 = kthread_run(thread1_func, NULL, "vdpa_map_unmap");
    thread2 = kthread_run(thread2_func, NULL, "vdpa_set_asid");

    return 0;
}

---

I keep thinking that this is a race condition in the hypothetical vdpa core or the module that runs this code, not solvable in VDUSE. We need to add barriers at this level, not VDUSE.

But I think it is a good idea to add some code in VDUSE to mitigate it. Maybe just adding the expected vq queue index to vdev->map->map_page and vdev->map->unmap_page? This way we can issue an error if the race happens.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Instead
of trying to do synchronization, maybe we can simply fail set_group_asid if DRIVER_OK is set.

Quoted 8 lines
Quoted 4 lines

That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.

Did you mean the SVQ may update the group asid while DRIVER_OK is set? If yes, we need to fix that.

Not at this moment.

Quoted 93 lines
Quoted 70 lines
Quoted 70 lines
Quoted 27 lines

    return NULL;
}
int main() {
    pthread_t thread0, thread1;
    int fd = open("/dev/vhost-vdpa-0", ...)

    pthread_create(&thread0, NULL, thread0_func, (void *)(intptr_t)fd);
    pthread_create(&thread1, NULL, thread1_func, (void *)(intptr_t)fd);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    return EXIT_SUCCESS;
}

---

We need something to synchronize at userland level here, filling the TODOs.

We can replace the hypothetical VHOST_VDPA_VQ_GROUP_DMA_MAP and _UNMAP with access to vqs, and the result is the same: The userland application is the one that needs to serialize the access from thread0 if it wants predictive outcome against the accesses from thread1. There is no way to do it at vduse level.

Quoted 5 lines

We need some syncrhonization to avoid malicious or buggy userland to mess things, that's for sure. So DMA_MAP and DMA_UNMAP does not half update the iotlb tree. And I'll send the next version with rwlock, protecting as much as possible. But I want to make clear that it will not avoid the race you describe here.

See above.

Quoted 34 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

But if you mean we depend on the IOTLB to guard against this, I'm fine, but let's document why we don't need it and how the IOTLB layer can help to eliminate such risk.

No, I'm not happy about letting iotlb lock to protect this too as they're at different levels actually: One is protecting iotlb trees modifications while they're being read and the other is protecting the ASID assignment to different vq groups. To reuse them means a modification in any tree blocks the change of vq group ASID, for example.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Anyhow, tracking and failing seems to be more robust.

I'm not sure I get this. If a DMA read starts in one ASID and then QEMU changes the ASID of the vq group, do you prefer it to fail rather than continue reading from the original ASID?

If possible, it would be better.

It's hard for me to think in some way that does not add a lot of overhead or it is very complex. But it is good to know this is acceptable.

Quoted 5 lines
Quoted 2 lines

It seems hard to
communicate that the ASID has changed to the DMA operation callback.

Maybe we can encode this into iova.

I'm adding the full rwlock, but can you expand on your idea on this?

Quoted 7 lines
Quoted 4 lines

Encode the asid to the upper bits of IOVA, so when doing dma_unmap() we can compare the group/as with the one that is encoded in dma_addr. If it differs, warn or bug.

But the ASID is already a parameter in the dma_unmap. I thought you meant to encode in the device's memory read.

To avoid unmap a address that is not belong to the this ASID.

Quoted 10 lines
Quoted 10 lines

If we can make sure there's no set_group_asid() when DRIVER_OK is set, it's not a must then (or could be treated as a kind of hardening).

Quoted 3 lines

It would be great to have it documented in case we need future optimizations.

Thanks

Thanks

#21 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 576 lines · Eugenio Perez Martin
Quoted 525 lines · Jason Wang
Quoted 523 lines · Eugenio Perez Martin
Quoted 521 lines · Jason Wang
Quoted 519 lines · Eugenio Perez Martin
Quoted 517 lines · Jason Wang
Quoted 350 lines · Eugenio Perez Martin
Quoted 348 lines · Jason Wang
Quoted 344 lines · Eugenio Perez Martin
Quoted 308 lines · Jason Wang
Quoted 186 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

Add support for assigning Address Space Identifiers (ASIDs) to each VQ
group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the
domain that each vq address point is also protected by domain_lock.

Maybe it's better to document what is protected by RCU and how.

I added the _rcu annotation but I can expand it for sure. I can also modify the commit message.

Quoted 4 lines

More below.

Quoted 2 lines

Acked-by: Jason Wang <[email protected]>

I forgot to remove this, my bad!

Quoted 148 lines
Quoted 148 lines

Signed-off-by: Eugenio Pérez <[email protected]>
---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic
context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to
justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/
(MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD
ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in
vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z)
return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions
(Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number
(Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower
value to reduce memory consumption, but vqs are already limited to
that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with
VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between
ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first
part of the struct is the same.
---
drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
include/uapi/linux/vduse.h | 53 ++++-
2 files changed, 314 insertions(+), 109 deletions(-)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 97be04f73fbf..ff95ed56f22d 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -11,6 +11,7 @@
#include "linux/virtio_net.h"
#include <linux/init.h>
#include <linux/module.h>
+#include <linux/rcupdate.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/eventfd.h>
@@ -41,6 +42,7 @@

#define VDUSE_DEV_MAX (1U << MINORBITS)
#define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
#define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
#define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
struct mm_struct *mm;
};

+struct vduse_as {
+ struct vduse_iova_domain *domain;
+ struct vduse_umem *umem;
+ struct mutex mem_lock;
+};
+
struct vduse_vq_group {
+ struct vduse_as *as __rcu;
struct vduse_dev *dev;
};

@@ -94,7 +103,7 @@ struct vduse_dev {
struct vduse_vdpa *vdev;
struct device *dev;
struct vduse_virtqueue **vqs;
- struct vduse_iova_domain *domain;
+ struct vduse_as *as;
char *name;
struct mutex lock;
spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
u32 vq_num;
u32 vq_align;
u32 ngroups;
- struct vduse_umem *umem;
+ u32 nas;
struct vduse_vq_group *groups;
- struct mutex mem_lock;
unsigned int bounce_size;
struct mutex domain_lock;
};
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
return vduse_dev_msg_sync(dev, &msg);
}

-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
u64 start, u64 last)
{
struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
return -EINVAL;

msg.req.type = VDUSE_UPDATE_IOTLB;
- msg.req.iova.start = start;
- msg.req.iova.last = last;
+ if (dev->api_version < VDUSE_API_VERSION_1) {
+ msg.req.iova.start = start;
+ msg.req.iova.last = last;
+ } else {
+ msg.req.iova_v2.start = start;
+ msg.req.iova_v2.last = last;
+ msg.req.iova_v2.asid = asid;
+ }

return vduse_dev_msg_sync(dev, &msg);
}
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
return mask;
}

Quoted 13 lines
Quoted 10 lines

+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+ unsigned int group, unsigned int asid)
+{
+ /*
+ * Two concurrent updates to this pointer are valid as they cannot
+ * point to an invalid region. It is ok for them to race as long as
+ * the readers see a consistent state through RCU.
+ */
+ rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

I'd expect at least a synchronize_rcu() here to wait for the read is done?

Quoted 3 lines

What's the use? The only thing left here is to return from vduse_set_group_asid_nomsg, and we don't need to wait for readers here, do we?

See below.

Quoted 102 lines
Quoted 88 lines
Quoted 88 lines

+}
+
static void vduse_dev_reset(struct vduse_dev *dev)
{
int i;
- struct vduse_iova_domain *domain = dev->domain;

/* The coherent mappings are handled in vduse_dev_free_coherent() */
- if (domain && domain->bounce_map)
- vduse_domain_reset_bounce_map(domain);
+ for (i = 0; i < dev->nas; i++) {
+ struct vduse_iova_domain *domain = dev->as[i].domain;
+
+ if (domain && domain->bounce_map)
+ vduse_domain_reset_bounce_map(domain);
+ }
+
+ for (i = 0; i < dev->ngroups; i++)
+ vduse_set_group_asid_nomsg(dev, i, 0);

down_write(&dev->rwsem);

@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
return ret;
}

+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+ unsigned int asid)
+{
+ struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+ struct vduse_dev_msg msg = { 0 };
+ int r;
+
+ if (dev->api_version < VDUSE_API_VERSION_1 ||
+ group >= dev->ngroups || asid >= dev->nas)
+ return -EINVAL;
+
+ msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+ msg.req.vq_group_asid.group = group;
+ msg.req.vq_group_asid.asid = asid;
+
+ r = vduse_dev_msg_sync(dev, &msg);
+ if (r < 0)
+ return r;
+
+ vduse_set_group_asid_nomsg(dev, group, asid);
+ return 0;
+}
+
static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
struct vdpa_vq_state *state)
{
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
struct vduse_dev *dev = vdpa_to_vduse(vdpa);
int ret;

- ret = vduse_domain_set_map(dev->domain, iotlb);
+ ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
if (ret)
return ret;

- ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+ ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
if (ret) {
- vduse_domain_clear_map(dev->domain, iotlb);
+ vduse_domain_clear_map(dev->as[asid].domain, iotlb);
return ret;
}

@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
.get_vq_affinity = vduse_vdpa_get_vq_affinity,
.reset = vduse_vdpa_reset,
.set_map = vduse_vdpa_set_map,
+ .set_group_asid = vduse_set_group_asid,
.get_vq_map = vduse_get_vq_map,
.free = vduse_vdpa_free,
};
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
enum dma_data_direction dir)
{
struct vduse_dev *vdev;
+ struct vduse_as *as;
struct vduse_iova_domain *domain;

if (!token.group)
return;

Quoted 13 lines
Quoted 8 lines

vdev = token.group->dev;
- domain = vdev->domain;
-
+ rcu_read_lock();
+ as = rcu_dereference(token.group->as);
+ domain = as->domain;
+ rcu_read_unlock();
vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);

This is suspicious, at least we should do rcu_read_unlock() after vduse_domain_sync_single_for_device(), otherwise I don't see how RCU works.

Quoted 7 lines

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Using READ_ONCE/WRITE_ONCE() needs to make sure the ordering is handled correctly.

Quoted 9 lines

But I meant what happens if

[cpu0]rcu_read_lock()
[cpu0]as = rcu_dereference(token.group->as)
[cpu0]...
[cpu0]rcu_read_unlock()
[cpu1]rcu_assign_pointer(token.group->as)
[cpu0]vduse_domain_sync_single_for_device()

That should go ok. What I'm trying to protect here is the iterations in vduse_domain_sync_single_for_device -> vduse_domain_bounce.

I'm going to embed that function here in vduse_dev_sync_single_for_device and omit RCU and some details to make the point easier:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
read_lock(&token.group->as->domain);
while (size)

    map = token.group->as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = token_group->as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;

}
read_unlock(&token.group->as->domain);
}

Quoted 6 lines
Quoted 4 lines

Right, so I meant for rwlock like semantic (let's forget the sleeping here).

vduse_set_group_asid_nomsg() should use "write lock" so it must wait for the "read lock" to be done.

No, it doesn't need to wait as long as the reader part uses its own copy.

It probably won't crash but I meant if we have logic issues. For example, once set_group_asid() return, there should still be a pending DMA that is using the old as.

Quoted 9 lines
Quoted 4 lines

But this is not the logic that is
implemented in this patch as there's no synchronize_rcu() in the vduse_set_group_asid_nomsg().

We only set the pointer on the writer's side, we do nothing like freeing resources. Should we set the pointer before or after syncrhonize_rcu()? What do we need to do on the other side of syncrhonize_rcu()?

Usually we don't need special care on the read side. But as discussed, synchronize_rcu() is not a must but we need to explain why it is safe and I'm not sure Michael is fine with that.
If we just want to make sure the order of publish and read, we can switch to use smp_store_release() and smp_load_acqurie().

Quoted 6 lines
Quoted 5 lines

We need to explain why set_group_asid()
doesn't need to wait and if this is true, we probably don't need RCU but to make sure the load/store is atomic.

Quoted 7 lines

What about:

* It does not matter if other thread modify group->as as long as the reader uses the same as for all its operation. It performs a local copy for that reason.
* It does not matter if multiple threads modify group->as as long as the update is atomic.

See above reply.

Quoted 71 lines

?

Quoted 10 lines
Quoted 10 lines

Now, depending on the point where another execution thread changes token_group->as and how the compiler has chosen to generate the machine code, the outcome could be:
1) The domain read lock of one ASID is taken but the domain lock of another as is unlocked.
2) We iterate until iova is ok for the ASID we're handling, but not for the other one. So we access an invalid offset in bounce_maps[iova].

Quoted 9 lines
Quoted 4 lines

And I guess there are other possible outcomes too.

So I need to make sure that the pointer accesses in all vduse_domain_bounce is coherent.

I'm not sure I got here, but it looks like it accepts a domain parameter and is protected by the bounce lock so we are probably fine here?

The bounce lock only protects the iotlb tree, not the pointer to that iotlb tree.

Quoted 45 lines
Quoted 21 lines

I'm ok if it takes the one before the
concurrent call to vduse_set_group_asid_nomsg or the one after that, as the lifetime of all domains are bound to the device. But it cannot change in the middle of the operation:

vduse_dev_sync_single_for_device(union virtio_map token, dma_addr_t iova, size_t size, ...) {
as = token.group->as;
// Tell the compiler to never replace "as" by "token.group->as" after this. read_lock(&as->domain);
while (size)

    map = as->domain->bounce_maps[iova];
    sz = min_t(size_t, BOUNCE_MAP_SIZE, size);

    ...
    page = as->domain->bounce_maps
    addr = kmap_local_page(page);
    do_bounce(map->orig_phys, addr, sz, dir);
    kunmap_local(addr);
    size -= sz;
    iova += sz;
Quoted 8 lines

}
read_unlock(&as->domain);
}

That can be done in many ways. Probably the read_lock is already enough but it is not explicit that it is protecting token.group->as, and future changes could remove it. To me, RCU is the most clear way to do it, but even a volatile read (READ_ONCE?) would do.

I wonder if another group rwlock is sufficient here:

for set_group_as_id()

write_lock(&dev->groups[group].lock);
dev->groups[group].as = &dev->as[asid];
write_unlock(&dev->groups[group].lock);

for the case where we need defer as

read_lock(&dev->groups[group].lock);
as = dev->groups[group].as;
//using as
read_unlock(&dev->groups[group].lock);

Quoted 7 lines
Quoted 3 lines

If this works, we don't need to bother with thinking if the wait/synchronizre_rcu() is really needed or not?

A rwlock is sufficient but we need to modify the allocation code somehow. Also, I thought we wanted to avoid the overhead of taking the read lock in the DMA ops too.

Right, but it would always be a balance. We can make sure it works correctly first then do optimization on top.

Quoted 38 lines

Another disadvantage of the lock vs RCU or READ_ONCE is that the vq group ASID change needs to wait for the DMA operation to finish instead of just applying for the next DMA ops. Not like vq group ASID change would be in the hot path anyway, just pointing it out.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines

If this is not an issue, RCU is not a must, but please explain why. If this is an issue, we need to fix it.

It's basically a question that

Quoted 7 lines
Quoted 2 lines

1) should we need to wait for the DMA to be completed before assigning to the new as

I don't think so, it is valid to assign a new as and let the ongoing operation to continue. It is racy and the operation could fail, but the kernel just returns an error and doesn't access invalid memory or similar.

See below.

Quoted 5 lines
Quoted 4 lines

2) should we track the set_group_asid() for the group that has pending DMA to avoid potential issue

Quoted 6 lines
Quoted 1 line

No, the group will outlive the operation as it is bound to the device.

I meant e.g the DMA could be triggered by the device. For example, the device may try to trigger an interrupt when the kernel is trying to assign a new asid. So I wonder if guest can use this to poke Qemu's memory etc.

I'm not sure I get this point. If QEMU changes the ASID of the vq group sent to the guest the race does not matter anymore: it is explicitly opening the possibility from the guest to poke QEMU's memory unless the guest is totally paused.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Basically what I meant, assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

[1]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

cpu0 may read as1 while it wants as0 actually?

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Yes, kind of. That's my point: adding synchronization at vduse level does not fix it.

There is no way to do that call from vhost/vdpa or userland, as there is no way to get the AS of a vq group, only to set it. The closest thing is to add a cache at that level, but that implies to add mutithreading sync on that upper layer, either vhost/vdpa or userland, not in VDUSE.

[2]

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Probably.

Quoted 5 lines

From vhost/vdpa level, all mapping calls (.set_map, .dma_map, .dma_unmap) calls take the ASID directly, not the vq group. So the call to set_group_asid does not need to access the vq group.

[3]

Quoted 15 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

Now let's say that we add that vdpa_ops callback (and ioctls) that maps and unmap based on a vq_group. And all of the operations (dma_map, set_group_asid, and dma_unmap) are serialized by taking the same mutex. cpu0 still may dma_unmap over as0 if set_group_asid is not properly serialized at vhost/vdpa or userland level:

void* thread0_func(void* arg) {

struct {
        int vq_group = 0,
        int iova, size, perm, ...
    } s;
    int fd = (intptr_t)arg;

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_MAP, &s);
Quoted 8 lines
Quoted 5 lines

I don't get the semantic of VHOST_VDPA_VQ_GROUP_DMA_MAP. And it's probably wrong to allow userspace to map based on group.

We have VHOST_IOTLB_UPDATE which map pages based on as (asid) this is correct.

Yes, I said that at [2], in reply to your scenario at [1] :). Did I misunderstand?

Nope, it looks like it's me that misread your reply.

Quoted 286 lines
Quoted 30 lines

More below

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
    // TODO: Signal thread0 that it can proceed with SET_GROUP_ASID
    // TODO: Wait until thread0 complete SET_GROUP_ASID

    ioctl(fd, VHOST_VDPA_VQ_GROUP_DMA_UNMAP, &data);

    return NULL;
}
void* thread1_func(void* arg) {
    struct vhost_vring_state s = {
        .index = 0,
        .num = 1,
    };
    int fd = (int)(intptr_t)arg;

    // TODO: Wait until thread2 calls dma_map
Quoted 7 lines
Quoted 3 lines

This is something exactly rwlock or synchronize_rcu() can do? Or is this the charge of the vDPA parent to do that?

No, that's an userland synchronization problem that cannot be solved at kernel level. If we need to do something similar at vdpa core level (unlikely), we will need to synchronize at that level too.

Ok, basically I think we are talking about different things. That's fine.

Quoted 8 lines

If I understand you correctly, you mean we need to synchronize between IOTLB updating (IOTLB_UPDATE/IOTLB_INVALIDATE) and set_group_asid()?

This sounds unnecessary since:

1) IOTLB_UPDATE/IOTLB_INVALIDATE is updating the address space internal mappings
2) set_group_asid() is to assign an AS to a group

Right, we're on the same page here [3].

Quoted 43 lines
Quoted 35 lines

If we add an rwlock but don't implement the synchronization in the userland program marked as TODO, this sequence is also possible:

cpu0] DMA map ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Now we have two possibilities: either cpu0] DMA_UNMAP is called or cpu1] set_group_asid is called. The VDUSE module rwlock protects that they will not run at the same time, but we need to implement something at the userland level if we want a predictive outcome, marked as the TODO in the comments. If, by chance, the DMA unmap is the one that comes next, the AS updated is the 0:

cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock

If set_group_asid run first by chance, ASID 1 is the one that is updated:
cpu1] set_group_asid ioctl call in userland
-> Take write_lock in the vduse module
-> Update ASID of the VQ GROUP 0 to 1
-> unlock write_lock
cpu0] DMA unmap ioctl call in userland
-> Take read_lock in the vduse module
-> Update the IOTLB tree of ASID 0
-> unlock read_lock

Quoted 7 lines

On the other hand we have this version of the series that allows these actions to run at the same time. It just makes sure that the update of the IOTLB tree is coherent, by copying the vq group ASID value at one point in time and making sure it sticks to that ASID until the end of the set_map call. I'm not adding the synchronize_rcu call because we stated it should not be called from userland, but the outcome is similar.

Quoted 2 lines

I think we should figure out if VHOST_VDPA_VQ_GROUP_DMA_MAP is useful or not.

It's not. I just wanted to expand how I understood your scenario, but let me know if I missed something!

Quoted 80 lines
Quoted 72 lines

Let me put another example: It's like calling fdup() and write() from different threads over the same set of fds without userland synchronization to me. The kernel protects things like not write part of the content in one file and another part on the other file, but the content of the files at the end of the writes() is just not predictable. And the kernel just cannot make it predictable.

static int fd_a;
static int fd_b;

void *write_thread(void *arg) {
    const char *msg = "Writing to fd_b\n";
    write(fd_b, msg, strlen(msg));
    return NULL;
}
void *dup_thread(void *arg) {
    dup2(fd_a, fd_b);
    return NULL;
}
int main() {
    pthread_t writer, duper;

    fd_a = open("/tmp/a", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    fd_b = open("/tmp/b", O_WRONLY | O_CREAT | O_TRUNC, 0644);

    pthread_create(&writer, NULL, write_thread, NULL);
    pthread_create(&duper, NULL, dup_thread, NULL);

    pthread_join(writer, NULL);
    pthread_join(duper, NULL);

    close(fd_a);
    close(fd_b);

    return 0;
}

--

Quoted 11 lines

If it's the responsibility of the parent, it would be much harder for VDUSE as the datapath is implemented in usersapce via mmap() or umem_reg.

Looking at existing implementation:

- for mlx5e, it looks like it assumes the set_group_asid() works only without DRIVER_OK.
- for the simulator, it looks like it can synchronize with the datapath with the spinlock as datapath is emulated

Yes, the next version will use the rwlock spinlock. I just need to take out the allocation of the domain for it to be valid.

Quoted 16 lines
Quoted 2 lines

ioctl(fd, VHOST_VDPA_SET_GROUP_ASID, &s)
// TODO: Signal thread2 that can proceed with dma_unmap

But the issue I mention is that the, from the view of the vDPA bus:

1) it offers set_group_as_id()
2) it doesn't know if virtio-vdpa or vhost-vdpa is used

So theoretically, set_group_as_id() could happen between

dma_addr = dma_map();

and

dma_unmap(dma_adrr);

Quoted 7 lines
Quoted 1 line

But those two dma_addr refers to the different address space.

I don't get this, these calls take the ASID as the parameter, not the vq group. I thought this was by design, as telling what vq groups update seems way more difficult to me. Can you put an example of an userland application that has the race you describe with the existing ioctls?

Quoted 4 lines

Just to clarify, I for dma_map()/dma_unmap() this is not the UAPI part (and we don't have that). I basically mean the dma API which is used by virtio-vDPA.

So let's say that the virtio_vdpa driver is able to call .set_group_asid just to explain how much it is protected:

->vduse_dev_map_page and ->vduse_dev_unmap_page
In this series, rcu_dereference makes sure that only the stack copy of the domain pointer is used, but the next version the code will have a read_lock taken protecting that set_group_asid actions on token.group->as cannot run at the same time that the vduse_domain_map_page(...) or vduse_domain_unmap_page(...) function call run in cpu0:

static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs)

{
        struct vduse_iova_domain *domain;
        dma_addr_t r;

        if (!token.group)
                return DMA_MAPPING_ERROR;

        read_lock(&dev->groups[group].as_lock);
        domain = token.group->as->domain;
        r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);
        read_unlock(&dev->groups[group].as_lock);

        return r;
}

static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
size_t size, enum dma_data_direction dir, unsigned long attrs)

{
        struct vduse_iova_domain *domain;

        if (!token.group)
                return;

        read_lock(&dev->groups[group].as_lock);
        domain = token.group->as->domain;
        vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
        read_unlock(&dev->groups[group].as_lock);
}

---

-> vduse_set_group_asid will have the corresponding write_lock, so it cannot modify any vq group AS pointer while they're mapping or unmapping pages from a domain:
static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
unsigned int group, unsigned int asid)

{
        write_lock(&dev->groups[group].as_lock);
        dev->groups[group].as = &dev->as[asid];
        write_unlock(&dev->groups[group].as_lock);
}

---

Now your question:

assuming group0.as = as0

cpu0] dma_map(group0.as, addr, DMA_FROM_DEVICE)
cpu1] set_group_asid(group0.as, as1)
cpu0] dma_unmap(group0.as, addr, DMA_FROM_DEVICE)

cpu0 may read as1 while it wants as0 actually?
--

Yes, it can happen even with the rwlock or any other synchronization in VDUSE, as it does not order the vdpa core calls.

This is the code I have in mind to prove this, let me know if I'm missing something from your scenario:

static int thread1_func(void *data)

{
    vduse_dev_map_page(token, ...);
    printk(KERN_INFO "map_page called\n");

    // TODO: Need a way to know if the group ASID has changed to
    // know if this unmap is valid or not.
    vduse_dev_unmap_page(token, ...);
    printk(KERN_INFO "unmap_page called\n");

    return 0;
}

static int thread2_func(void *data)

{
    u32 vq_group = 0;
    u32 asid = 1;

    vdpa->ops->set_group_asid(global_vdpa, vq_group, asid);
    printk(KERN_INFO "set_group_asid called\n");

    return 0;
}

static int __init kernel_threads_init(void)

{
    struct task_struct *thread1, *thread2;

    thread1 = kthread_run(thread1_func, NULL, "vdpa_map_unmap");
    thread2 = kthread_run(thread2_func, NULL, "vdpa_set_asid");

    return 0;
}

---

Quoted 7 lines

I keep thinking that this is a race condition in the hypothetical vdpa core or the module that runs this code, not solvable in VDUSE. We need to add barriers at this level, not VDUSE.

But I think it is a good idea to add some code in VDUSE to mitigate it. Maybe just adding the expected vq queue index to vdev->map->map_page and vdev->map->unmap_page?

This seems to be more complicated than encoding asid to the iova.

Quoted 2 lines

This way we can issue
an error if the race happens.

As discussed, it could be avoided by disallowing the call to set_group_asid() when DRIVER_OK is set.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

Instead
of trying to do synchronization, maybe we can simply fail set_group_asid if DRIVER_OK is set.

That's a good possibility, especially since mlx5 already does it. There is ongoing work to enable dataplane SVQ dynamically without needing to reset the whole device, but we will need a new feature flag to know if the parent driver supports it.

Quoted 5 lines
Quoted 3 lines

Did you mean the SVQ may update the group asid while DRIVER_OK is set? If yes, we need to fix that.

Not at this moment.

Great.

Thanks

#22 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

In reply to Eugenio Perez Martin
Quoted 7 lines · Eugenio Perez Martin

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

generally rcu itself does not need ONCE macros. these are for funky lockless things, and rcu can be seen as a kind of lock, after all.

Signature · 1 line
MST
author maintainer
#23 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Eugenio Perez Martin

RCU is protecting that the address space pointer of the vq group is not modified concurrently with the access. Ideally, this should be a full lock, but just making sure that all accesses from the reader are coherent is enough. Userspace should expect nothing if it uses the map and modifies the vq group ASID at the same time anyway, but the kernel needs to be sure that it does not see intermediate states. TBH, we could move to a READ_ONCE / WRITE_ONCE, would that be more clear?

Quoted 4 lines

generally rcu itself does not need ONCE macros. these are for funky lockless things, and rcu can be seen as a kind of lock, after all.

Right, I meant to replace RCU by READ_ONCE / WRITE ONCE. But I also prefer RCU.

#24 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

In reply to Eugenio Pérez
kernel test robot noticed the following build warnings:

Hi Eugenio,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.18-rc5 next-20251113]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Eugenio-P-rez/vduse-add-v1-API-definition/20251113-200303
base:   linus/master
patch link:    https://lore.kernel.org/r/20251113115558.1277981-6-eperezma%40redhat.com
patch subject: [PATCH v9 5/6] vduse: add vq group asid support
config: arc-randconfig-001-20251114 (https://download.01.org/0day-ci/archive/20251114/[email protected]/config)
compiler: arc-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251114/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_sync_single_for_device':

>> drivers/vdpa/vdpa_user/vduse_dev.c:910:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~
   drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_sync_single_for_cpu':
   drivers/vdpa/vdpa_user/vduse_dev.c:929:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~
   drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_map_page':
   drivers/vdpa/vdpa_user/vduse_dev.c:949:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~
   drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_unmap_page':
   drivers/vdpa/vdpa_user/vduse_dev.c:971:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~
   drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_alloc_coherent':
   drivers/vdpa/vdpa_user/vduse_dev.c:989:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~
   drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_need_sync':
   drivers/vdpa/vdpa_user/vduse_dev.c:1034:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~
   drivers/vdpa/vdpa_user/vduse_dev.c: In function 'vduse_dev_max_mapping_size':
   drivers/vdpa/vdpa_user/vduse_dev.c:1061:20: warning: variable 'vdev' set but not used [-Wunused-but-set-variable]
     struct vduse_dev *vdev;
                       ^~~~

vim +/vdev +910 drivers/vdpa/vdpa_user/vduse_dev.c

c8a6153b6c59d9 Xie Yongji      2021-08-31  905  
1c14b0e4ba9883 Jason Wang      2025-09-24  906  static void vduse_dev_sync_single_for_device(union virtio_map token,
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19  907  					     dma_addr_t dma_addr, size_t size,
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19  908  					     enum dma_data_direction dir)
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19  909  {
f98606c92d03d8 Eugenio Pérez   2025-11-13 @910  	struct vduse_dev *vdev;
1e9ae6f6f2d50d Eugenio Pérez   2025-11-13  911  	struct vduse_as *as;
f98606c92d03d8 Eugenio Pérez   2025-11-13  912  	struct vduse_iova_domain *domain;
f98606c92d03d8 Eugenio Pérez   2025-11-13  913  
f98606c92d03d8 Eugenio Pérez   2025-11-13  914  	if (!token.group)
f98606c92d03d8 Eugenio Pérez   2025-11-13  915  		return;
f98606c92d03d8 Eugenio Pérez   2025-11-13  916  
f98606c92d03d8 Eugenio Pérez   2025-11-13  917  	vdev = token.group->dev;
1e9ae6f6f2d50d Eugenio Pérez   2025-11-13  918  	rcu_read_lock();
1e9ae6f6f2d50d Eugenio Pérez   2025-11-13  919  	as = rcu_dereference(token.group->as);
1e9ae6f6f2d50d Eugenio Pérez   2025-11-13  920  	domain = as->domain;
1e9ae6f6f2d50d Eugenio Pérez   2025-11-13  921  	rcu_read_unlock();
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19  922  	vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19  923  }
d7b4e3287ca3a7 Maxime Coquelin 2024-02-19  924  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#25 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

In reply to Eugenio Pérez
kernel test robot noticed the following build warnings:

Hi Eugenio,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.18-rc5 next-20251113]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Eugenio-P-rez/vduse-add-v1-API-definition/20251113-200303
base:   linus/master
patch link:    https://lore.kernel.org/r/20251113115558.1277981-6-eperezma%40redhat.com
patch subject: [PATCH v9 5/6] vduse: add vq group asid support
config: i386-randconfig-062-20251114 (https://download.01.org/0day-ci/archive/20251114/[email protected]/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251114/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

sparse warnings: (new ones prefixed by >>)

>> drivers/vdpa/vdpa_user/vduse_dev.c:462:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:462:9: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:462:9: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:919:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:919:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:919:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:938:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:938:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:938:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:959:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:959:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:959:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:980:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:980:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:980:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:1001:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:1001:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:1001:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:1025:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:1025:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:1025:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:1044:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:1044:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:1044:14: sparse:    struct vduse_as *
   drivers/vdpa/vdpa_user/vduse_dev.c:1071:14: sparse: sparse: incompatible types in comparison expression (different address spaces):
   drivers/vdpa/vdpa_user/vduse_dev.c:1071:14: sparse:    struct vduse_as [noderef] __rcu *
   drivers/vdpa/vdpa_user/vduse_dev.c:1071:14: sparse:    struct vduse_as *
>> drivers/vdpa/vdpa_user/vduse_dev.c:1074:23: sparse: sparse: dereference of noderef expression

vim +462 drivers/vdpa/vdpa_user/vduse_dev.c

452
453 /* Force set the asid to a vq group without a message to the VDUSE device */
454 static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
455 unsigned int group, unsigned int asid)
456 {
457 /*
458 * Two concurrent updates to this pointer are valid as they cannot
459 * point to an invalid region. It is ok for them to race as long as
460 * the readers see a consistent state through RCU.
461 */

 > 462		rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);

463 }
464

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#26 lore raw

PATCH v9 5/6 vduse: add vq group asid support

In reply to Eugenio Pérez
Quoted 409 lines

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group.  This enables mapping each group into a distinct memory space.
Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++--------- include/uapi/linux/vduse.h         |  53 ++++-

 2 files changed, 314 insertions(+), 109 deletions(-)

drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
  #include "linux/virtio_net.h"
  #include <linux/init.h>
  #include <linux/module.h>
+#include <linux/rcupdate.h>
  #include <linux/cdev.h>
  #include <linux/device.h>
  #include <linux/eventfd.h>
@@ -41,6 +42,7 @@


 #define VDUSE_DEV_MAX (1U << MINORBITS) #define VDUSE_DEV_MAX_GROUPS 0xffff +#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024) #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024) #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)

drivers/vdpa/vdpa_user/vduse_dev.c +0 -0
@@ -86,7 +88,14 @@ struct vduse_umem {
          struct mm_struct *mm;
  };

drivers/vdpa/vdpa_user/vduse_dev.c +6 -0
+struct vduse_as {
+        struct vduse_iova_domain *domain;
+        struct vduse_umem *umem;
+        struct mutex mem_lock;
+};
+

 struct vduse_vq_group {
+        struct vduse_as *as __rcu;
         struct vduse_dev *dev;
 };

drivers/vdpa/vdpa_user/vduse_dev.c +2 -3
@@ -94,7 +103,7 @@ struct vduse_dev {
          struct vduse_vdpa *vdev;
          struct device *dev;
          struct vduse_virtqueue **vqs;
-        struct vduse_iova_domain *domain;
+        struct vduse_as *as;
          char *name;
          struct mutex lock;
          spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
          u32 vq_num;
          u32 vq_align;
          u32 ngroups;
-        struct vduse_umem *umem;
+        u32 nas;
          struct vduse_vq_group *groups;
-        struct mutex mem_lock;
          unsigned int bounce_size;
          struct mutex domain_lock;
  };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
          return vduse_dev_msg_sync(dev, &msg);
  }

drivers/vdpa/vdpa_user/vduse_dev.c +1 -1
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,

                                   u64 start, u64 last) {
         struct vduse_dev_msg msg = { 0 };

drivers/vdpa/vdpa_user/vduse_dev.c +0 -0
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                  return -EINVAL;


         msg.req.type = VDUSE_UPDATE_IOTLB;
-        msg.req.iova.start = start;
-        msg.req.iova.last = last;
+        if (dev->api_version < VDUSE_API_VERSION_1) {
+                msg.req.iova.start = start;
+                msg.req.iova.last = last;
+        } else {
+                msg.req.iova_v2.start = start;
+                msg.req.iova_v2.last = last;
+                msg.req.iova_v2.asid = asid;
+        }

         return vduse_dev_msg_sync(dev, &msg); }

drivers/vdpa/vdpa_user/vduse_dev.c +0 -0
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
          return mask;
  }

drivers/vdpa/vdpa_user/vduse_dev.c +12 -0
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                       unsigned int group, unsigned int asid)
+{
+        /*
+         * Two concurrent updates to this pointer are valid as they cannot
+         * point to an invalid region.  It is ok for them to race as long as
+         * the readers see a consistent state through RCU.
+         */
+        rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);
+}
+

 static void vduse_dev_reset(struct vduse_dev *dev)
 {
         int i;
-        struct vduse_iova_domain *domain = dev->domain;

         /* The coherent mappings are handled in vduse_dev_free_coherent() */
-        if (domain && domain->bounce_map)
-                vduse_domain_reset_bounce_map(domain);
+        for (i = 0; i < dev->nas; i++) {
+                struct vduse_iova_domain *domain = dev->as[i].domain;

drivers/vdpa/vdpa_user/vduse_dev.c +7 -0
+
+                if (domain && domain->bounce_map)
+                        vduse_domain_reset_bounce_map(domain);
+        }
+
+        for (i = 0; i < dev->ngroups; i++)
+                vduse_set_group_asid_nomsg(dev, i, 0);


         down_write(&dev->rwsem);

drivers/vdpa/vdpa_user/vduse_dev.c +0 -0
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
          return ret;
  }

drivers/vdpa/vdpa_user/vduse_dev.c +23 -0
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                                unsigned int asid)
+{
+        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+        struct vduse_dev_msg msg = { 0 };
+        int r;
+
+        if (dev->api_version < VDUSE_API_VERSION_1 ||
+            group >= dev->ngroups || asid >= dev->nas)
+                return -EINVAL;
+
+        msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+        msg.req.vq_group_asid.group = group;
+        msg.req.vq_group_asid.asid = asid;
+
+        r = vduse_dev_msg_sync(dev, &msg);
+        if (r < 0)
+                return r;
+
+        vduse_set_group_asid_nomsg(dev, group, asid);
+        return 0;
+}
+

 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx, struct vdpa_vq_state *state) {

drivers/vdpa/vdpa_user/vduse_dev.c +0 -0
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
          struct vduse_dev *dev = vdpa_to_vduse(vdpa);
          int ret;


-        ret = vduse_domain_set_map(dev->domain, iotlb);
+        ret = vduse_domain_set_map(dev->as[asid].domain, iotlb); if (ret)
                 return ret;

-        ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+        ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX); if (ret) {
-                vduse_domain_clear_map(dev->domain, iotlb);
+                vduse_domain_clear_map(dev->as[asid].domain, iotlb); return ret;
         }

drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
          .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
          .reset                        = vduse_vdpa_reset,
          .set_map                = vduse_vdpa_set_map,
+        .set_group_asid                = vduse_set_group_asid,
          .get_vq_map                = vduse_get_vq_map,
          .free                        = vduse_vdpa_free,
  };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                               enum dma_data_direction dir)
  {
          struct vduse_dev *vdev;
+        struct vduse_as *as;
          struct vduse_iova_domain *domain;


         if (!token.group)
                 return;

         vdev = token.group->dev;
-        domain = vdev->domain;
-
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock(); vduse_domain_sync_single_for_device(domain, dma_addr, size, dir); }

drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                               enum dma_data_direction dir)
  {
          struct vduse_dev *vdev;
+        struct vduse_as *as;
          struct vduse_iova_domain *domain;


         if (!token.group)
                 return;

         vdev = token.group->dev;
-        domain = vdev->domain;
-
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock(); vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir); }

drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                       unsigned long attrs)
  {
          struct vduse_dev *vdev;
+        struct vduse_as *as;
          struct vduse_iova_domain *domain;
+        dma_addr_t r;


         if (!token.group)
                 return DMA_MAPPING_ERROR;

         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);
-        return vduse_domain_map_page(domain, page, offset, size, dir, attrs);
+        return r;
 }

 static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,

drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
@@ -901,21 +969,25 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
                                   unsigned long attrs)
  {
          struct vduse_dev *vdev;
+        struct vduse_as *as;
          struct vduse_iova_domain *domain;


         if (!token.group)
                 return;

         vdev = token.group->dev;
-        domain = vdev->domain;
-
-        return vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs); }

 static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size, dma_addr_t *dma_addr, gfp_t flag) {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
         unsigned long iova;
         void *addr;

drivers/vdpa/vdpa_user/vduse_dev.c +0 -0
@@ -925,13 +997,14 @@ static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,
                  return NULL;


         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
         addr = vduse_domain_alloc_coherent(domain, size, (dma_addr_t *)&iova, flag);
-        if (!addr)
-                return NULL;
-
-        *dma_addr = (dma_addr_t)iova;
+        if (addr)
+                *dma_addr = (dma_addr_t)iova;

         return addr;
 }

drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
@@ -941,29 +1014,39 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,
                                      unsigned long attrs)
  {
          struct vduse_dev *vdev;
+        struct vduse_as *as;
          struct vduse_iova_domain *domain;


         if (!token.group)
                 return;

         vdev = token.group->dev;
-        domain = vdev->domain;
-
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
         vduse_domain_free_coherent(domain, size, vaddr, dma_addr, attrs);
+        mutex_unlock(&vdev->domain_lock);
 }

 static bool vduse_dev_need_sync(union virtio_map token, dma_addr_t dma_addr) {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
+        size_t bounce_size;

         if (!token.group)
                 return false;

         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        bounce_size = domain->bounce_size;

-        return dma_addr < domain->bounce_size;
+        return dma_addr < bounce_size;
 }

 static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)

drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
@@ -976,15 +1059,22 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
  static size_t vduse_dev_max_mapping_size(union virtio_map token)
  {
          struct vduse_dev *vdev;
+        struct vduse_as *as;
          struct vduse_iova_domain *domain;
+        size_t bounce_size;


         if (!token.group)
                 return 0;

Quoted 8 lines


         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        domain = token.group->as->domain;

This line should be removed?

Thanks,
Yongji

author maintainer
#27 lore raw

Re: [PATCH v9 5/6] vduse: add vq group asid support

Quoted 411 lines · 谢永吉
Quoted 411 lines

Add support for assigning Address Space Identifiers (ASIDs) to each VQ group. This enables mapping each group into a distinct memory space.

Now that the driver can change ASID in the middle of operation, the domain that each vq address point is also protected by domain_lock.

---
v9:
* Replace mutex with RCU, as the vdpa map_ops can run from atomic context.

v8:
* Revert the mutex to rwlock change, it needs proper profiling to justify it.

v7:
* Take write lock in the error path (Jason).

v6:
* Make vdpa_dev_add use gotos for error handling (MST).
* s/(dev->api_version < 1) ?/(dev->api_version < VDUSE_API_VERSION_1) ?/ (MST).
* Fix struct name not matching in the doc.

v5:
* Properly return errno if copy_to_user returns >0 in VDUSE_IOTLB_GET_FD ioctl (Jason).
* Properly set domain bounce size to divide equally between nas (Jason).
* Exclude "padding" member from the only >V1 members in vduse_dev_request.

v4:
* Divide each domain bounce size between the device bounce size (Jason).
* revert unneeded addr = NULL assignment (Jason)
* Change if (x && (y || z)) return to if (x) { if (y) return; if (z) return; } (Jason)
* Change a bad multiline comment, using @ caracter instead of * (Jason).
* Consider config->nas == 0 as a fail (Jason).

v3:
* Get the vduse domain through the vduse_as in the map functions (Jason).
* Squash with the patch creating the vduse_as struct (Jason).
* Create VDUSE_DEV_MAX_AS instead of comparing agains a magic number (Jason)

v2:
* Convert the use of mutex to rwlock.

RFC v3:
* Increase VDUSE_MAX_VQ_GROUPS to 0xffff (Jason). It was set to a lower value to reduce memory consumption, but vqs are already limited to that value and userspace VDUSE is able to allocate that many vqs.
* Remove TODO about merging VDUSE_IOTLB_GET_FD ioctl with VDUSE_IOTLB_GET_INFO.
* Use of array_index_nospec in VDUSE device ioctls.
* Embed vduse_iotlb_entry into vduse_iotlb_entry_v2.
* Move the umem mutex to asid struct so there is no contention between ASIDs.

RFC v2:
* Make iotlb entry the last one of vduse_iotlb_entry_v2 so the first part of the struct is the same.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 370 ++++++++++++++++++++---------
 include/uapi/linux/vduse.h         |  53 ++++-
 2 files changed, 314 insertions(+), 109 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +115 -30
index 97be04f73fbf..ff95ed56f22d 100644
@@ -11,6 +11,7 @@
 #include "linux/virtio_net.h"
 #include <linux/init.h>
 #include <linux/module.h>
+#include <linux/rcupdate.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
 #include <linux/eventfd.h>
@@ -41,6 +42,7 @@
 
 #define VDUSE_DEV_MAX (1U << MINORBITS)
 #define VDUSE_DEV_MAX_GROUPS 0xffff
+#define VDUSE_DEV_MAX_AS 0xffff
 #define VDUSE_MAX_BOUNCE_SIZE (1024 * 1024 * 1024)
 #define VDUSE_MIN_BOUNCE_SIZE (1024 * 1024)
 #define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024)
@@ -86,7 +88,14 @@ struct vduse_umem {
         struct mm_struct *mm;
 };
 
+struct vduse_as {
+        struct vduse_iova_domain *domain;
+        struct vduse_umem *umem;
+        struct mutex mem_lock;
+};
+
 struct vduse_vq_group {
+        struct vduse_as *as __rcu;
         struct vduse_dev *dev;
 };
 
@@ -94,7 +103,7 @@ struct vduse_dev {
         struct vduse_vdpa *vdev;
         struct device *dev;
         struct vduse_virtqueue **vqs;
-        struct vduse_iova_domain *domain;
+        struct vduse_as *as;
         char *name;
         struct mutex lock;
         spinlock_t msg_lock;
@@ -122,9 +131,8 @@ struct vduse_dev {
         u32 vq_num;
         u32 vq_align;
         u32 ngroups;
-        struct vduse_umem *umem;
+        u32 nas;
         struct vduse_vq_group *groups;
-        struct mutex mem_lock;
         unsigned int bounce_size;
         struct mutex domain_lock;
 };
@@ -314,7 +322,7 @@ static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
         return vduse_dev_msg_sync(dev, &msg);
 }
 
-static int vduse_dev_update_iotlb(struct vduse_dev *dev,
+static int vduse_dev_update_iotlb(struct vduse_dev *dev, u32 asid,
                                   u64 start, u64 last)
 {
         struct vduse_dev_msg msg = { 0 };
@@ -323,8 +331,14 @@ static int vduse_dev_update_iotlb(struct vduse_dev *dev,
                 return -EINVAL;
 
         msg.req.type = VDUSE_UPDATE_IOTLB;
-        msg.req.iova.start = start;
-        msg.req.iova.last = last;
+        if (dev->api_version < VDUSE_API_VERSION_1) {
+                msg.req.iova.start = start;
+                msg.req.iova.last = last;
+        } else {
+                msg.req.iova_v2.start = start;
+                msg.req.iova_v2.last = last;
+                msg.req.iova_v2.asid = asid;
+        }
 
         return vduse_dev_msg_sync(dev, &msg);
 }
@@ -436,14 +450,32 @@ static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
         return mask;
 }
 
+/* Force set the asid to a vq group without a message to the VDUSE device */
+static void vduse_set_group_asid_nomsg(struct vduse_dev *dev,
+                                       unsigned int group, unsigned int asid)
+{
+        /*
+         * Two concurrent updates to this pointer are valid as they cannot
+         * point to an invalid region.  It is ok for them to race as long as
+         * the readers see a consistent state through RCU.
+         */
+        rcu_assign_pointer(dev->groups[group].as, &dev->as[asid]);
+}
+
 static void vduse_dev_reset(struct vduse_dev *dev)
 {
         int i;
-        struct vduse_iova_domain *domain = dev->domain;
 
         /* The coherent mappings are handled in vduse_dev_free_coherent() */
-        if (domain && domain->bounce_map)
-                vduse_domain_reset_bounce_map(domain);
+        for (i = 0; i < dev->nas; i++) {
+                struct vduse_iova_domain *domain = dev->as[i].domain;
+
+                if (domain && domain->bounce_map)
+                        vduse_domain_reset_bounce_map(domain);
+        }
+
+        for (i = 0; i < dev->ngroups; i++)
+                vduse_set_group_asid_nomsg(dev, i, 0);
 
         down_write(&dev->rwsem);
 
@@ -623,6 +655,29 @@ static union virtio_map vduse_get_vq_map(struct vdpa_device *vdpa, u16 idx)
         return ret;
 }
 
+static int vduse_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
+                                unsigned int asid)
+{
+        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
+        struct vduse_dev_msg msg = { 0 };
+        int r;
+
+        if (dev->api_version < VDUSE_API_VERSION_1 ||
+            group >= dev->ngroups || asid >= dev->nas)
+                return -EINVAL;
+
+        msg.req.type = VDUSE_SET_VQ_GROUP_ASID;
+        msg.req.vq_group_asid.group = group;
+        msg.req.vq_group_asid.asid = asid;
+
+        r = vduse_dev_msg_sync(dev, &msg);
+        if (r < 0)
+                return r;
+
+        vduse_set_group_asid_nomsg(dev, group, asid);
+        return 0;
+}
+
 static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
                                 struct vdpa_vq_state *state)
 {
@@ -794,13 +849,13 @@ static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
         int ret;
 
-        ret = vduse_domain_set_map(dev->domain, iotlb);
+        ret = vduse_domain_set_map(dev->as[asid].domain, iotlb);
         if (ret)
                 return ret;
 
-        ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
+        ret = vduse_dev_update_iotlb(dev, asid, 0ULL, ULLONG_MAX);
         if (ret) {
-                vduse_domain_clear_map(dev->domain, iotlb);
+                vduse_domain_clear_map(dev->as[asid].domain, iotlb);
                 return ret;
         }
 
@@ -843,6 +898,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
         .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
         .reset                        = vduse_vdpa_reset,
         .set_map                = vduse_vdpa_set_map,
+        .set_group_asid                = vduse_set_group_asid,
         .get_vq_map                = vduse_get_vq_map,
         .free                        = vduse_vdpa_free,
 };
@@ -852,14 +908,17 @@ static void vduse_dev_sync_single_for_device(union virtio_map token,
                                              enum dma_data_direction dir)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
 
         if (!token.group)
                 return;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
-
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
         vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);
 }
 
@@ -868,14 +927,17 @@ static void vduse_dev_sync_single_for_cpu(union virtio_map token,
                                              enum dma_data_direction dir)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
 
         if (!token.group)
                 return;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
-
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
         vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
 }
 
@@ -885,15 +947,21 @@ static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
                                      unsigned long attrs)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
+        dma_addr_t r;
 
         if (!token.group)
                 return DMA_MAPPING_ERROR;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        r = vduse_domain_map_page(domain, page, offset, size, dir, attrs);
 
-        return vduse_domain_map_page(domain, page, offset, size, dir, attrs);
+        return r;
 }
 
 static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
@@ -901,21 +969,25 @@ static void vduse_dev_unmap_page(union virtio_map token, dma_addr_t dma_addr,
                                  unsigned long attrs)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
 
         if (!token.group)
                 return;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
-
-        return vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
 }
 
 static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,
                                       dma_addr_t *dma_addr, gfp_t flag)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
         unsigned long iova;
         void *addr;
@@ -925,13 +997,14 @@ static void *vduse_dev_alloc_coherent(union virtio_map token, size_t size,
                 return NULL;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
         addr = vduse_domain_alloc_coherent(domain, size,
                                            (dma_addr_t *)&iova, flag);
-        if (!addr)
-                return NULL;
-
-        *dma_addr = (dma_addr_t)iova;
+        if (addr)
+                *dma_addr = (dma_addr_t)iova;
 
         return addr;
 }
@@ -941,29 +1014,39 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,
                                     unsigned long attrs)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
 
         if (!token.group)
                 return;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
-
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
         vduse_domain_free_coherent(domain, size, vaddr, dma_addr, attrs);
+        mutex_unlock(&vdev->domain_lock);
 }
 
 static bool vduse_dev_need_sync(union virtio_map token, dma_addr_t dma_addr)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
+        size_t bounce_size;
 
         if (!token.group)
                 return false;
 
         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        bounce_size = domain->bounce_size;
 
-        return dma_addr < domain->bounce_size;
+        return dma_addr < bounce_size;
 }
 
 static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
@@ -976,15 +1059,22 @@ static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
 static size_t vduse_dev_max_mapping_size(union virtio_map token)
 {
         struct vduse_dev *vdev;
+        struct vduse_as *as;
         struct vduse_iova_domain *domain;
+        size_t bounce_size;
 
         if (!token.group)
                 return 0;
 
Quoted 10 lines

drivers/vdpa/vdpa_user/vduse_dev.c static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr) · line 1074

Quoted 7 lines
drivers/vdpa/vdpa_user/vduse_dev.c +5 -1
         vdev = token.group->dev;
-        domain = vdev->domain;
+        rcu_read_lock();
+        as = rcu_dereference(token.group->as);
+        domain = as->domain;
+        rcu_read_unlock();
+        domain = token.group->as->domain;

This line should be removed?

Totally, thank you very much for the catch!

author maintainer
#28 lore raw

PATCH v9 6/6 vduse: bump version number

In reply to Eugenio Pérez

Finalize the series by advertising VDUSE API v1 support to userspace.

Now that all required infrastructure for v1 (ASIDs, VQ groups, update_iotlb_v2) is in place, VDUSE devices can opt in to the new features.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +2 -2
index ff95ed56f22d..f5a1bc027ecb 100644
@@ -2188,7 +2188,7 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
 			break;
 
 		ret = -EINVAL;
-		if (api_version > VDUSE_API_VERSION)
+		if (api_version > VDUSE_API_VERSION_1)
 			break;
 
 		ret = 0;
@@ -2255,7 +2255,7 @@ static int vduse_open(struct inode *inode, struct file *file)
 	if (!control)
 		return -ENOMEM;
 
-	control->api_version = VDUSE_API_VERSION;
+	control->api_version = VDUSE_API_VERSION_1;
 	file->private_data = control;
 
 	return 0;

--
2.51.1