Skip to contentSkip to conversation

vduse: Add support for reconnection

PATCH In review Cindy Lu 3 participants 19 messages Apr 12, 2024 – Apr 25, 2024

Subsystems (2)
lore
Share & tools
lore

Cindy Lu

author
#0 lore raw

v5 cover vduse: Add support for reconnection

Here is the reconnect support in vduse

Kernel will allocate pages for reconnection.
Userspace needs to use mmap to map the memory to userspace and use these pages to save the reconnect information.

test passd in vduse+dpdk-testpmd

change in V2
1. Address the comments from v1
2. Add the document for reconnect process

change in V3
1. Move the vdpa_vq_state to the uAPI. vduse will use this to synchronize the vq info between the kernel and userspace app.
2. Add a new ioctl VDUSE_DEV_GET_CONFIG. userspace app use this to get config space
3. Rewrite the commit message.
4. Only save the address for the page address and remove the index.
5. remove the ioctl VDUSE_GET_RECONNECT_INFO, userspace app will use uAPI VDUSE_RECONNCT_MMAP_SIZE to mmap
6. Rewrite the document for the reconnect process to make it clearer.

change in v4
1. Change the number of map pages to VQ numbers. UserSpace APP can define and maintain the structure for saving reconnection information in userspace. The kernel will not maintain this information.
2. Rewrite the document for the reconnect process to make it clearer.
3. add the new ioctl for VDUSE_DEV_GET_CONFIG/VDUSE_DEV_GET_STATUS

change in V5
1. update the Documentation for vduse reconnection
2. merge the patch from Dan Carpenter <[email protected]> vduse: Fix off by one in vduse_dev_mmap()
3. fix the wrong comment in the code

Cindy Lu (5):
vduse: Add new ioctl VDUSE_DEV_GET_CONFIG
vduse: Add new ioctl VDUSE_DEV_GET_STATUS
vduse: Add function to get/free the pages for reconnection
vduse: Add file operation for mmap
Documentation: Add reconnect process for VDUSE

 Documentation/userspace-api/vduse.rst |  41 +++++++++
 drivers/vdpa/vdpa_user/vduse_dev.c    | 125 ++++++++++++++++++++++++++
 include/uapi/linux/vduse.h            |   5 ++
 3 files changed, 171 insertions(+)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v5 1/5 vduse: Add new ioctl VDUSE_DEV_GET_CONFIG

The ioctl VDUSE_DEV_GET_CONFIG is used by the Userspace App to get the device configuration space.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 21 +++++++++++++++++++++
 include/uapi/linux/vduse.h         |  3 +++
 2 files changed, 24 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +21 -0
index 1c1d71d69026..ab246da27616 100644
@@ -1368,6 +1368,27 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 		ret = 0;
 		break;
 	}
+	case VDUSE_DEV_GET_CONFIG: {
+		struct vduse_config_data config;
+		unsigned long size = offsetof(struct vduse_config_data, buffer);
+
+		ret = -EFAULT;
+		if (copy_from_user(&config, argp, size))
+			break;
+
+		ret = -EINVAL;
+		if (config.offset > dev->config_size || config.length == 0 ||
+		    config.length > dev->config_size - config.offset)
+			break;
+
+		if (copy_to_user(argp + size, dev->config + config.offset,
+				 config.length)) {
+			ret = -EFAULT;
+			break;
+		}
+		ret = 0;
+		break;
+	}
 	default:
 		ret = -ENOIOCTLCMD;
 		break;
include/uapi/linux/vduse.h +3 -0
index 11bd48c72c6c..125d7529d91b 100644
@@ -350,4 +350,7 @@ struct vduse_dev_response {
 	};
 };
 
+/* get device configuration space */
+#define VDUSE_DEV_GET_CONFIG _IOR(VDUSE_BASE, 0x1b, struct vduse_config_data)
+
 #endif /* _UAPI_VDUSE_H_ */
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH v5 2/5 vduse: Add new ioctl VDUSE_DEV_GET_STATUS

In reply to Cindy Lu

The ioctl VDUSE_DEV_GET_STATUS is used by the Userspace App to get the device status

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
 include/uapi/linux/vduse.h         | 2 ++
 2 files changed, 9 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +7 -0
index ab246da27616..ef3c9681941e 100644
@@ -1389,6 +1389,13 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 		ret = 0;
 		break;
 	}
+
+	case VDUSE_DEV_GET_STATUS:
+		/*
+		 * Returns the status read from device
+		 */
+		ret = put_user(dev->status, (u8 __user *)argp);
+		break;
 	default:
 		ret = -ENOIOCTLCMD;
 		break;
include/uapi/linux/vduse.h +2 -0
index 125d7529d91b..f501173a9d69 100644
@@ -353,4 +353,6 @@ struct vduse_dev_response {
 /* get device configuration space */
 #define VDUSE_DEV_GET_CONFIG _IOR(VDUSE_BASE, 0x1b, struct vduse_config_data)
 
+#define VDUSE_DEV_GET_STATUS _IOR(VDUSE_BASE, 0x1c, __u8)
+
 #endif /* _UAPI_VDUSE_H_ */
Signature · 1 line
2.43.0
author
#3 lore raw

PATCH v5 3/5 vduse: Add function to get/free the pages for reconnection

In reply to Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +40 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
 	int irq_effective_cpu;
 	struct cpumask irq_affinity;
 	struct kobject kobj;
+	unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
 	vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+	unsigned long vaddr = 0;
+	struct vduse_virtqueue *vq;
+
+	for (int i = 0; i < dev->vq_num; i++) {
+		/*page 0~ vq_num save the reconnect info for vq*/
+		vq = dev->vqs[i];
+		vaddr = get_zeroed_page(GFP_KERNEL);
+		if (vaddr == 0)
+			return -ENOMEM;
+
+		vq->vdpa_reconnect_vaddr = vaddr;
+	}
+
+	return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+	struct vduse_virtqueue *vq;
+
+	for (int i = 0; i < dev->vq_num; i++) {
+		vq = dev->vqs[i];
+
+		if (vq->vdpa_reconnect_vaddr)
+			free_page(vq->vdpa_reconnect_vaddr);
+		vq->vdpa_reconnect_vaddr = 0;
+	}
+
+	return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 			    unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
 		mutex_unlock(&dev->lock);
 		return -EBUSY;
 	}
+	vduse_free_reconnnect_info_mem(dev);
+
 	dev->connected = true;
 	mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
 	if (ret)
 		goto err_vqs;
+	ret = vduse_alloc_reconnnect_info_mem(dev);
+	if (ret < 0)
+		goto err_mem;
 
 	__module_get(THIS_MODULE);
 
 	return 0;
 err_vqs:
 	device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+	vduse_free_reconnnect_info_mem(dev);
 err_dev:
 	idr_remove(&vduse_idr, dev->minor);
 err_idr:
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 26 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
 	int irq_effective_cpu;
 	struct cpumask irq_affinity;
 	struct kobject kobj;
+	unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
 	vq->irq_effective_cpu = curr_cpu;

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +9 -0
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+	unsigned long vaddr = 0;
+	struct vduse_virtqueue *vq;
+
+	for (int i = 0; i < dev->vq_num; i++) {
+		/*page 0~ vq_num save the reconnect info for vq*/
+		vq = dev->vqs[i];
+		vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Quoted 55 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -0
+		if (vaddr == 0)
+			return -ENOMEM;
+
+		vq->vdpa_reconnect_vaddr = vaddr;
+	}
+
+	return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+	struct vduse_virtqueue *vq;
+
+	for (int i = 0; i < dev->vq_num; i++) {
+		vq = dev->vqs[i];
+
+		if (vq->vdpa_reconnect_vaddr)
+			free_page(vq->vdpa_reconnect_vaddr);
+		vq->vdpa_reconnect_vaddr = 0;
+	}
+
+	return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
 			    unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
 		mutex_unlock(&dev->lock);
 		return -EBUSY;
 	}
+	vduse_free_reconnnect_info_mem(dev);
+
 	dev->connected = true;
 	mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
 	if (ret)
 		goto err_vqs;
+	ret = vduse_alloc_reconnnect_info_mem(dev);
+	if (ret < 0)
+		goto err_mem;
 
 	__module_get(THIS_MODULE);
 
 	return 0;
 err_vqs:
 	device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+	vduse_free_reconnnect_info_mem(dev);
 err_dev:
 	idr_remove(&vduse_idr, dev->minor);
 err_idr:
Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs
2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#6 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 46 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Quoted 4 lines

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

No idea what this means.

Quoted 3 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

And I don't see why that has to live in the kernel tree either.

Quoted 64 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#7 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 48 lines · Michael S. Tsirkin
Quoted 46 lines · Jason Wang
Quoted 44 lines · Michael S. Tsirkin
Quoted 26 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +9 -0
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Quoted 6 lines
Quoted 4 lines

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Quoted 6 lines
Quoted 4 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

And I don't see why that has to live in the kernel tree either.

I can't find a better place, any idea?

Thanks

Quoted 66 lines
Quoted 65 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#8 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 51 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Good point. So the motivation here is to:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

1) be self contained, no dependency for high speed persist data storage like tmpfs

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Are we worried about systems without tmpfs now?

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

And I don't see why that has to live in the kernel tree either.

I can't find a better place, any idea?

Thanks

Well anywhere on github really. with libvhost-user maybe? It's harmless enough in Documentation
if you like but ties you to the kernel release cycle in a way that is completely unnecessary.

Quoted 66 lines
Quoted 66 lines
Quoted 65 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#9 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 56 lines · Michael S. Tsirkin
Quoted 54 lines · Jason Wang
Quoted 53 lines · Michael S. Tsirkin
Quoted 50 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Are we worried about systems without tmpfs now?

Yes.

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

And I don't see why that has to live in the kernel tree either.

I can't find a better place, any idea?

Quoted 7 lines
Quoted 1 line

Thanks

Well anywhere on github really. with libvhost-user maybe? It's harmless enough in Documentation
if you like but ties you to the kernel release cycle in a way that is completely unnecessary.

Ok.

Thanks

Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

author
#10 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 82 lines · Jason Wang
Quoted 56 lines · Michael S. Tsirkin
Quoted 54 lines · Jason Wang
Quoted 53 lines · Michael S. Tsirkin
Quoted 50 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Are we worried about systems without tmpfs now?

Yes.

Quoted 14 lines
Quoted 12 lines
Quoted 8 lines
Quoted 6 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

And I don't see why that has to live in the kernel tree either.

I can't find a better place, any idea?

Thanks

Quoted 9 lines
Quoted 4 lines

Well anywhere on github really. with libvhost-user maybe? It's harmless enough in Documentation
if you like but ties you to the kernel release cycle in a way that is completely unnecessary.

Ok.

Thanks

Sure, got it. Do we need to withdraw all the series? maybe we can keep the patch for support ioctl to get status and configure?

thanks
cindy

Quoted 69 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#11 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 89 lines · Cindy Lu
Quoted 87 lines · Jason Wang
Quoted 56 lines · Michael S. Tsirkin
Quoted 54 lines · Jason Wang
Quoted 53 lines · Michael S. Tsirkin
Quoted 50 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Are we worried about systems without tmpfs now?

Yes.

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

Quoted 5 lines
Quoted 1 line

And I don't see why that has to live in the kernel tree either.

I can't find a better place, any idea?

Thanks

Well anywhere on github really. with libvhost-user maybe? It's harmless enough in Documentation
if you like but ties you to the kernel release cycle in a way that is completely unnecessary.

Quoted 6 lines
Quoted 4 lines

Ok.

Thanks

Sure, got it. Do we need to withdraw all the series? maybe we can keep the patch for support ioctl to get status and configure?

We can leave the mmap part for future, the rest is still useful unless I miss anything.

Thanks

Quoted 74 lines

thanks
cindy

Quoted 69 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#12 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

In reply to Jason Wang
Quoted 59 lines · Jason Wang
Quoted 58 lines · Michael S. Tsirkin
Quoted 56 lines · Jason Wang
Quoted 54 lines · Michael S. Tsirkin
Quoted 46 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Quoted 4 lines

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

No idea what this means.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Are we worried about systems without tmpfs now?

Yes.

Why? Who ships these?

Quoted 91 lines
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

2) standardize the format in uAPI which allows reconnection from arbitrary userspace, unfortunately, such effort was removed in new versions

And I don't see why that has to live in the kernel tree either.

I can't find a better place, any idea?

Quoted 7 lines
Quoted 1 line

Thanks

Well anywhere on github really. with libvhost-user maybe? It's harmless enough in Documentation
if you like but ties you to the kernel release cycle in a way that is completely unnecessary.

Ok.

Thanks

Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines

If the above doesn't make sense, we don't need to offer those pages by VDUSE.

Thanks

Quoted 59 lines
Quoted 58 lines
drivers/vdpa/vdpa_user/vduse_dev.c +30 -1
+             if (vaddr == 0)
+                     return -ENOMEM;
+
+             vq->vdpa_reconnect_vaddr = vaddr;
+     }
+
+     return 0;
+}
+
+static int vduse_free_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             vq = dev->vqs[i];
+
+             if (vq->vdpa_reconnect_vaddr)
+                     free_page(vq->vdpa_reconnect_vaddr);
+             vq->vdpa_reconnect_vaddr = 0;
+     }
+
+     return 0;
+}
 
 static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
                          unsigned long arg)
@@ -1672,6 +1705,8 @@ static int vduse_destroy_dev(char *name)
              mutex_unlock(&dev->lock);
              return -EBUSY;
      }
+     vduse_free_reconnnect_info_mem(dev);
+
      dev->connected = true;
      mutex_unlock(&dev->lock);
 
@@ -1855,12 +1890,17 @@ static int vduse_create_dev(struct vduse_dev_config *config,
      ret = vduse_dev_init_vqs(dev, config->vq_align, config->vq_num);
      if (ret)
              goto err_vqs;
+     ret = vduse_alloc_reconnnect_info_mem(dev);
+     if (ret < 0)
+             goto err_mem;
 
      __module_get(THIS_MODULE);
 
      return 0;
 err_vqs:
      device_destroy(&vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
+err_mem:
+     vduse_free_reconnnect_info_mem(dev);
 err_dev:
      idr_remove(&vduse_idr, dev->minor);
 err_idr:
--

2.43.0

#13 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 62 lines · Michael S. Tsirkin
Quoted 59 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 46 lines · Jason Wang
Quoted 44 lines · Michael S. Tsirkin
Quoted 26 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +1 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +9 -0
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Quoted 6 lines
Quoted 4 lines

Good point. So the motivation here is to:

1) be self contained, no dependency for high speed persist data storage like tmpfs

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Are we worried about systems without tmpfs now?

Yes.

Why? Who ships these?

Not sure, but it could be disabled or unmounted. I'm not sure make VDUSE depends on TMPFS is a good idea.

Thanks

#14 lore raw

Re: [PATCH v5 3/5] vduse: Add function to get/free the pages for reconnection

Quoted 68 lines · Jason Wang
Quoted 67 lines · Michael S. Tsirkin
Quoted 65 lines · Jason Wang
Quoted 63 lines · Michael S. Tsirkin
Quoted 51 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Cindy Lu

Add the function vduse_alloc_reconnnect_info_mem
and vduse_alloc_reconnnect_info_mem
These functions allow vduse to allocate and free memory for reconnection information. The amount of memory allocated is vq_num pages.
Each VQS will map its own page where the reconnection information will be saved

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 40 ++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +8 -0
index ef3c9681941e..2da659d5f4a8 100644
@@ -65,6 +65,7 @@ struct vduse_virtqueue {
      int irq_effective_cpu;
      struct cpumask irq_affinity;
      struct kobject kobj;
+     unsigned long vdpa_reconnect_vaddr;
 };
 
 struct vduse_dev;
@@ -1105,6 +1106,38 @@ static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
 
      vq->irq_effective_cpu = curr_cpu;
 }
+static int vduse_alloc_reconnnect_info_mem(struct vduse_dev *dev)
+{
+     unsigned long vaddr = 0;
+     struct vduse_virtqueue *vq;
+
+     for (int i = 0; i < dev->vq_num; i++) {
+             /*page 0~ vq_num save the reconnect info for vq*/
Quoted 8 lines

drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq) · line 1117

Quoted 2 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2 -0
+             vq = dev->vqs[i];
+             vaddr = get_zeroed_page(GFP_KERNEL);

I don't get why you insist on stealing kernel memory for something that is just used by userspace to store data for its own use. Userspace does not lack ways to persist data, for example, create a regular file anywhere in the filesystem.

Good point. So the motivation here is to:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

1) be self contained, no dependency for high speed persist data storage like tmpfs

No idea what this means.

I mean a regular file may slow down the datapath performance, so usually the application will try to use tmpfs and other which is a dependency for implementing the reconnection.

Are we worried about systems without tmpfs now?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes.

Why? Who ships these?

Not sure, but it could be disabled or unmounted. I'm not sure make VDUSE depends on TMPFS is a good idea.

Thanks

Don't disable or unmount it then?
The use-case needs to be much clearer if we are adding a way for userspace to pin kernel memory for unlimited time.

Signature · 1 line
MST
author
#15 lore raw

PATCH v5 4/5 vduse: Add file operation for mmap

In reply to Cindy Lu

Add the operation for mmap, This function will be used by the user space application to map the pages to the user space.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 57 ++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
drivers/vdpa/vdpa_user/vduse_dev.c +57 -0
index 2da659d5f4a8..7abe0c17cf0e 100644
@@ -1465,6 +1465,61 @@ static struct vduse_dev *vduse_dev_get_from_minor(int minor)
 	return dev;
 }
 
+static vm_fault_t vduse_vm_fault(struct vm_fault *vmf)
+{
+	struct vduse_dev *dev = vmf->vma->vm_file->private_data;
+	struct vm_area_struct *vma = vmf->vma;
+	u16 index = vma->vm_pgoff;
+	struct vduse_virtqueue *vq;
+	unsigned long vaddr;
+
+	/* index 0~vq_number-1 page reserved for virtqueue state*/
+	vq = dev->vqs[index];
+	vaddr = vq->vdpa_reconnect_vaddr;
+	if (remap_pfn_range(vma, vmf->address & PAGE_MASK,
+			    PFN_DOWN(virt_to_phys((void *)vaddr)), PAGE_SIZE,
+			    vma->vm_page_prot))
+		return VM_FAULT_SIGBUS;
+	return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct vduse_vm_ops = {
+	.fault = vduse_vm_fault,
+};
+
+static int vduse_dev_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	struct vduse_dev *dev = file->private_data;
+	unsigned long vaddr = 0;
+	unsigned long index = vma->vm_pgoff;
+	struct vduse_virtqueue *vq;
+
+	if (vma->vm_end - vma->vm_start != PAGE_SIZE)
+		return -EINVAL;
+	if ((vma->vm_flags & VM_SHARED) == 0)
+		return -EINVAL;
+	if (index >= dev->vq_num)
+		return -EINVAL;
+
+	index = array_index_nospec(index, dev->vq_num);
+	vq = dev->vqs[index];
+	vaddr = vq->vdpa_reconnect_vaddr;
+	if (vaddr == 0)
+		return -EOPNOTSUPP;
+
+	if (virt_to_phys((void *)vaddr) & (PAGE_SIZE - 1))
+		return -EINVAL;
+
+	/* Check if the Userspace App mapped the correct size */
+	if (vma->vm_end - vma->vm_start != PAGE_SIZE)
+		return -EOPNOTSUPP;
+
+	vm_flags_set(vma, VM_DONTEXPAND);
+	vma->vm_ops = &vduse_vm_ops;
+
+	return 0;
+}
+
 static int vduse_dev_open(struct inode *inode, struct file *file)
 {
 	int ret;
@@ -1497,6 +1552,8 @@ static const struct file_operations vduse_dev_fops = {
 	.unlocked_ioctl	= vduse_dev_ioctl,
 	.compat_ioctl	= compat_ptr_ioctl,
 	.llseek		= noop_llseek,
+	.mmap		= vduse_dev_mmap,
+
 };
 
 static ssize_t irq_cb_affinity_show(struct vduse_virtqueue *vq, char *buf)
Signature · 1 line
2.43.0
author
#16 lore raw

PATCH v5 5/5 Documentation: Add reconnect process for VDUSE

In reply to Cindy Lu

Add a document explaining the reconnect process, including what the Userspace App needs to do and how it works with the kernel.

---
 Documentation/userspace-api/vduse.rst | 41 +++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
Documentation/userspace-api/vduse.rst +41 -0
index bdb880e01132..7faa83462e78 100644
@@ -231,3 +231,44 @@ able to start the dataplane processing as follows:
    after the used ring is filled.
 
 For more details on the uAPI, please see include/uapi/linux/vduse.h.
+
+HOW VDUSE devices reconnection works
+------------------------------------
+1. What is reconnection?
+
+   When the userspace application loads, it should establish a connection
+   to the vduse kernel device. Sometimes,the userspace application exists,
+   and we want to support its restart and connect to the kernel device again
+
+2. How can I support reconnection in a userspace application?
+
+2.1 During initialization, the userspace application should first verify the
+    existence of the device "/dev/vduse/vduse_name".
+    If it doesn't exist, it means this is the first-time for connection. goto step 2.2
+    If it exists, it means this is a reconnection, and we should goto step 2.3
+
+2.2 Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
+    /dev/vduse/control.
+    When ioctl(VDUSE_CREATE_DEV) is called, kernel allocates memory for
+    the reconnect information. The total memory size is PAGE_SIZE*vq_mumber.
+
+2.3 Check if the information is suitable for reconnect
+    If this is reconnection :
+    Before attempting to reconnect, The userspace application needs to use the
+    ioctl(VDUSE_DEV_GET_CONFIG, VDUSE_DEV_GET_STATUS, VDUSE_DEV_GET_FEATURES...)
+    to get the information from kernel.
+    Please review the information and confirm if it is suitable to reconnect.
+
+2.4 Userspace application needs to mmap the memory to userspace
+    The userspace application requires mapping one page for every vq. These pages
+    should be used to save vq-related information during system running. Additionally,
+    the application must define its own structure to store information for reconnection.
+
+2.5 Completed the initialization and running the application.
+    While the application is running, it is important to store relevant information
+    about reconnections in mapped pages. When calling the ioctl VDUSE_VQ_GET_INFO to
+    get vq information, it's necessary to check whether it's a reconnection. If it is
+    a reconnection, the vq-related information must be get from the mapped pages.
+
+2.6 When the Userspace application exits, it is necessary to unmap all the
+    pages for reconnection
Signature · 1 line
2.43.0
#17 lore raw

Re: [PATCH v5 5/5] Documentation: Add reconnect process for VDUSE

Quoted 14 lines · Cindy Lu

Add a document explaining the reconnect process, including what the Userspace App needs to do and how it works with the kernel.

---
 Documentation/userspace-api/vduse.rst | 41 +++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
Documentation/userspace-api/vduse.rst +0 -0
index bdb880e01132..7faa83462e78 100644

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 240

Quoted 10 lines
Documentation/userspace-api/vduse.rst +7 -0
@@ -231,3 +231,44 @@ able to start the dataplane processing as follows:
    after the used ring is filled.
 
 For more details on the uAPI, please see include/uapi/linux/vduse.h.
+
+HOW VDUSE devices reconnection works
+------------------------------------
+1. What is reconnection?
+
+   When the userspace application loads, it should establish a connection
+   to the vduse kernel device. Sometimes,the userspace application exists,

I guess you meant "exists"? If yes, it should be better to say "exits unexpectedly"

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 243

Quoted 3 lines
Documentation/userspace-api/vduse.rst +3 -0
+   and we want to support its restart and connect to the kernel device again
+
+2. How can I support reconnection in a userspace application?

Better to say "How reconnection is supported"?

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 253

Quoted 10 lines
Documentation/userspace-api/vduse.rst +10 -0
+
+2.1 During initialization, the userspace application should first verify the
+    existence of the device "/dev/vduse/vduse_name".
+    If it doesn't exist, it means this is the first-time for connection. goto step 2.2
+    If it exists, it means this is a reconnection, and we should goto step 2.3
+
+2.2 Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
+    /dev/vduse/control.
+    When ioctl(VDUSE_CREATE_DEV) is called, kernel allocates memory for
+    the reconnect information. The total memory size is PAGE_SIZE*vq_mumber.

I think we need to mention that this should be part of the previous "VDUSE devices are created as follows"?

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 260

Quoted 7 lines
Documentation/userspace-api/vduse.rst +7 -0
+
+2.3 Check if the information is suitable for reconnect
+    If this is reconnection :
+    Before attempting to reconnect, The userspace application needs to use the
+    ioctl(VDUSE_DEV_GET_CONFIG, VDUSE_DEV_GET_STATUS, VDUSE_DEV_GET_FEATURES...)
+    to get the information from kernel.
+    Please review the information and confirm if it is suitable to reconnect.

Need to define "review" here and how to decide if it is not suitable to reconnect.

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows:

Quoted 4 lines
Documentation/userspace-api/vduse.rst +4 -0
+
+2.4 Userspace application needs to mmap the memory to userspace
+    The userspace application requires mapping one page for every vq. These pages
+    should be used to save vq-related information during system running.

Not a native speaker, but it looks better with

"should be used by the userspace to store virtqueue specific information".

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows:

Quoted 6 lines

Additionally,

Documentation/userspace-api/vduse.rst +5 -0
+    the application must define its own structure to store information for reconnection.
+
+2.5 Completed the initialization and running the application.
+    While the application is running, it is important to store relevant information
+    about reconnections in mapped pages.

I think we need some link/code examples to demonstrate what needs to be stored.

Quoted 2 lines

When calling the ioctl VDUSE_VQ_GET_INFO to
+ get vq information, it's necessary to check whether it's a reconnection.

Better with some examples of codes.

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 274

Quoted 5 lines

If it is

Documentation/userspace-api/vduse.rst +4 -0
+    a reconnection, the vq-related information must be get from the mapped pages.
+
+2.6 When the Userspace application exits, it is necessary to unmap all the
+    pages for reconnection

This seems to be unnecessary, for example there could be an unexpected exit.

Thanks

Quoted 3 lines

--
2.43.0

author
#18 lore raw

Re: [PATCH v5 5/5] Documentation: Add reconnect process for VDUSE

Quoted 90 lines · Jason Wang
Quoted 14 lines · Cindy Lu

Add a document explaining the reconnect process, including what the Userspace App needs to do and how it works with the kernel.

---
 Documentation/userspace-api/vduse.rst | 41 +++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
Documentation/userspace-api/vduse.rst +0 -0
index bdb880e01132..7faa83462e78 100644

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 240

Quoted 10 lines
Documentation/userspace-api/vduse.rst +7 -0
@@ -231,3 +231,44 @@ able to start the dataplane processing as follows:
    after the used ring is filled.
 
 For more details on the uAPI, please see include/uapi/linux/vduse.h.
+
+HOW VDUSE devices reconnection works
+------------------------------------
+1. What is reconnection?
+
+   When the userspace application loads, it should establish a connection
+   to the vduse kernel device. Sometimes,the userspace application exists,

I guess you meant "exists"? If yes, it should be better to say "exits unexpectedly"

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 243

Quoted 3 lines
Documentation/userspace-api/vduse.rst +3 -0
+   and we want to support its restart and connect to the kernel device again
+
+2. How can I support reconnection in a userspace application?

Better to say "How reconnection is supported"?

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 253

Quoted 10 lines
Documentation/userspace-api/vduse.rst +10 -0
+
+2.1 During initialization, the userspace application should first verify the
+    existence of the device "/dev/vduse/vduse_name".
+    If it doesn't exist, it means this is the first-time for connection. goto step 2.2
+    If it exists, it means this is a reconnection, and we should goto step 2.3
+
+2.2 Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
+    /dev/vduse/control.
+    When ioctl(VDUSE_CREATE_DEV) is called, kernel allocates memory for
+    the reconnect information. The total memory size is PAGE_SIZE*vq_mumber.

I think we need to mention that this should be part of the previous "VDUSE devices are created as follows"?

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 260

Quoted 7 lines
Documentation/userspace-api/vduse.rst +7 -0
+
+2.3 Check if the information is suitable for reconnect
+    If this is reconnection :
+    Before attempting to reconnect, The userspace application needs to use the
+    ioctl(VDUSE_DEV_GET_CONFIG, VDUSE_DEV_GET_STATUS, VDUSE_DEV_GET_FEATURES...)
+    to get the information from kernel.
+    Please review the information and confirm if it is suitable to reconnect.

Need to define "review" here and how to decide if it is not suitable to reconnect.

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows:

Quoted 4 lines
Documentation/userspace-api/vduse.rst +4 -0
+
+2.4 Userspace application needs to mmap the memory to userspace
+    The userspace application requires mapping one page for every vq. These pages
+    should be used to save vq-related information during system running.

Not a native speaker, but it looks better with

"should be used by the userspace to store virtqueue specific information".

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows:

Quoted 6 lines

Additionally,

Documentation/userspace-api/vduse.rst +5 -0
+    the application must define its own structure to store information for reconnection.
+
+2.5 Completed the initialization and running the application.
+    While the application is running, it is important to store relevant information
+    about reconnections in mapped pages.

I think we need some link/code examples to demonstrate what needs to be stored.

Quoted 2 lines

When calling the ioctl VDUSE_VQ_GET_INFO to
+ get vq information, it's necessary to check whether it's a reconnection.

Better with some examples of codes.

Documentation/userspace-api/vduse.rst able to start the dataplane processing as follows: · line 274

Quoted 5 lines

If it is

Documentation/userspace-api/vduse.rst +4 -0
+    a reconnection, the vq-related information must be get from the mapped pages.
+
+2.6 When the Userspace application exits, it is necessary to unmap all the
+    pages for reconnection
Quoted 4 lines

This seems to be unnecessary, for example there could be an unexpected exit.

Thanks

Thanks Jason, I will send a new version
Thanks
Cindy

Quoted 4 lines
Quoted 3 lines

--
2.43.0