Skip to contentSkip to conversation

Add queue ready message to VDUSE

PATCH Superseded Eugenio Pérez 3 participants 14 messages Feb 10 – Mar 3 virtio core

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eugenio Pérez

author maintainer
#0 lore raw

v2 cover Add queue ready message to VDUSE

This series introduces a new VDUSE message for VDUSE userland instance to detect when a VirtQueue (VQ) is enabled, replacing the polling.

VirtIO net devices' dataplane is started after the control virtqueue so QEMU can apply the configuration in the destination of a Live Migration. Without this feature, the VDUSE instance must poll the VQs to check when (and if) a VQ has been enabled.

This series also implements VDUSE feature flags allowing the VDUSE devices to opt-in to the VQ ready message. Devices that opt-in to this feature will receive explicit notifications when a VQ is ready. Devices that do not set this flag remain unaffected, ensuring backward compatibility without indefinitely incrementing API versions.

The VDUSE features is a 64 bit bitmap for simplicity, the same way as vhost and vhost-net started. It can be extended as a flexible array of bits when we reach so many features, but it seems unlikely at this point.

Error cases tested:
* Call VDUSE_GET_FEATURES without get the API VERSION (so API == 0) and with API VERSION set to 1 with VDUSE_SET_API_VERSION (-EINVAL returned from VDUSE_GET_FEATURES ioctl).
* Try to create a device with config->features different than 0 without fetch the api version (so API == 0) and with API_VERSION set to 1 (-EINVAL returned from VDUSE_CREATE_DEV ioctl).
* Test regular initialization of single queue devices with V2 with and without VDUSE_F_QUEUE_READY set in device->config.
* Test expected behavior when VDUSE userland instance returns VDUSE_REQ_RESULT_FAILED from VDUSE_SET_VQ_READY message.
* Repeat all the tests with multiqueue devices, by reverting 56e71885b0349 ("vduse: Temporarily fail if control queue feature requested").

This series depends on
https://lore.kernel.org/lkml/[email protected]/

v2:
* Fix comment of vduse_dev_request.vq_ready
* Set vq_ready before sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.
* Return -EINVAL if control ioctl called with version < 2, so userland visible reply is kept (Jason).

Eugenio Pérez (5):
vduse: store control device pointer
vduse: Add API v2 definition
vduse: add VDUSE_GET_FEATURES ioctl
vduse: advertise API V2 support
vduse: add F_QUEUE_READY feature

 drivers/vdpa/vdpa_user/vduse_dev.c | 66 +++++++++++++++++++++++++++---
 include/uapi/linux/vduse.h         | 29 ++++++++++++-
 2 files changed, 88 insertions(+), 7 deletions(-)

--
2.53.0

author maintainer
#1 lore raw

PATCH v2 1/5 vduse: store control device pointer

This helps log the errors in next patches. The alternative is to perform a linear search for it with class_find_device_by_devt(class, devt), as device_destroy do for cleaning.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +4 -4
index 405d59610f76..d1da7c15d98b 100644
@@ -164,6 +164,7 @@ static DEFINE_IDR(vduse_idr);
 
 static dev_t vduse_major;
 static struct cdev vduse_ctrl_cdev;
+static const struct device *vduse_ctrl_dev;
 static struct cdev vduse_cdev;
 static struct workqueue_struct *vduse_irq_wq;
 static struct workqueue_struct *vduse_irq_bound_wq;
@@ -2396,7 +2397,6 @@ static void vduse_mgmtdev_exit(void)
 static int vduse_init(void)
 {
 	int ret;
-	struct device *dev;
 
 	ret = class_register(&vduse_class);
 	if (ret)
@@ -2413,9 +2413,9 @@ static int vduse_init(void)
 	if (ret)
 		goto err_ctrl_cdev;
 
-	dev = device_create(&vduse_class, NULL, vduse_major, NULL, "control");
-	if (IS_ERR(dev)) {
-		ret = PTR_ERR(dev);
+	vduse_ctrl_dev = device_create(&vduse_class, NULL, vduse_major, NULL, "control");
+	if (IS_ERR(vduse_ctrl_dev)) {
+		ret = PTR_ERR(vduse_ctrl_dev);
 		goto err_device;
 	}
 
Signature · 1 line
2.53.0
author maintainer
#2 lore raw

PATCH v2 2/5 vduse: Add API v2 definition

In reply to Eugenio Pérez

Introduce the definition for VDUSE API V2. This version serves as a gateway for feature negotiation.

The kernel uses this version to determine if the userspace device supports feature flags. Devices that do not explicitly negotiate API V2 will be blocked from querying available VDUSE features, ensuring backward compatibility.

The next patches implement the new feature incrementally, only enabling the VDUSE device to set the V2 API version by the end of the series.

---
 include/uapi/linux/vduse.h | 4 ++++
 1 file changed, 4 insertions(+)
include/uapi/linux/vduse.h +4 -0
index 361eea511c21..27832d46084c 100644
@@ -14,6 +14,10 @@
 
 #define VDUSE_API_VERSION_1	1
 
+/* Features support */
+
+#define VDUSE_API_VERSION_2	2
+
 /*
  * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
  * This is used for future extension.
Signature · 1 line
2.53.0
author maintainer
#3 lore raw

PATCH v2 3/5 vduse: add VDUSE_GET_FEATURES ioctl

In reply to Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
 			 sizeof(config->reserved)))
 		return false;
 
+	if (api_version < VDUSE_API_VERSION_2) {
+		if (config->vduse_features) {
+			dev_dbg(vduse_ctrl_dev,
+				"config->vduse_features with version %llu",
+				api_version);
+			return false;
+		}
+	} else {
+		if (config->vduse_features & ~vduse_features)
+			return false;
+	}
+
+
 	if (api_version < VDUSE_API_VERSION_1 &&
 	    (config->ngroups || config->nas))
 		return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
 		ret = vduse_destroy_dev(name);
 		break;
 	}
+	case VDUSE_GET_FEATURES:
+		if (control->api_version < VDUSE_API_VERSION_2) {
+			dev_dbg(vduse_ctrl_dev,
+				"VDUSE_GET_FEATURES ioctl with version %llu",
+				control->api_version);
+			ret = -EINVAL;
+			break;
+		}
+
+		ret = put_user(vduse_features, (u64 __user *)argp);
+		break;
+
 	default:
 		ret = -EINVAL;
 		break;
include/uapi/linux/vduse.h +6 -1
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space
@@ -53,7 +54,8 @@ struct vduse_dev_config {
 	__u32 vq_align;
 	__u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
 	__u32 nas; /* if VDUSE_API_VERSION >= 1 */
-	__u32 reserved[11];
+	__u64 vduse_features;
+	__u32 reserved[9];
 	__u32 config_size;
 	__u8 config[];
 };
@@ -67,6 +69,9 @@ struct vduse_dev_config {
  */
 #define VDUSE_DESTROY_DEV	_IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
 
+/* Get the VDUSE supported features */
+#define VDUSE_GET_FEATURES	_IOR(VDUSE_BASE, 0x04, __u64)
+
 /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
 
 /**
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Thanks

Quoted 17 lines
include/uapi/linux/vduse.h +4 -1
+       __u32 reserved[9];
        __u32 config_size;
        __u8 config[];
 };
@@ -67,6 +69,9 @@ struct vduse_dev_config {
  */
 #define VDUSE_DESTROY_DEV      _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
 
+/* Get the VDUSE supported features */
+#define VDUSE_GET_FEATURES     _IOR(VDUSE_BASE, 0x04, __u64)
+
 /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
 
 /**
--

2.53.0

author maintainer
#5 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

Quoted 97 lines · Jason Wang
Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

Quoted 4 lines

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

How do you know if you can call VDUSE_GET_FEATURES and VDUSE_SET_FEATURES then? vduse_ioctl does not even return -ENOIOCMD but -EINVAL if you call the equivalent of VDUSE_GET_FEATURES on current Linux master. Or should the VDUSE instance assume that if the ioctl returns -EINVAL the kernel does not support them?

#6 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

Quoted 104 lines · Eugenio Perez Martin
Quoted 97 lines · Jason Wang
Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

Quoted 4 lines

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Quoted 5 lines

How do you know if you can call VDUSE_GET_FEATURES and VDUSE_SET_FEATURES then? vduse_ioctl does not even return -ENOIOCMD but -EINVAL if you call the equivalent of VDUSE_GET_FEATURES on current Linux master. Or should the VDUSE instance assume that if the ioctl returns -EINVAL the kernel does not support them?

If we go that way, yes.

Thanks

Quoted 1 line
author maintainer
#7 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

Quoted 107 lines · Jason Wang
Quoted 104 lines · Eugenio Perez Martin
Quoted 101 lines · Jason Wang
Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Quoted 8 lines
Quoted 5 lines

How do you know if you can call VDUSE_GET_FEATURES and VDUSE_SET_FEATURES then? vduse_ioctl does not even return -ENOIOCMD but -EINVAL if you call the equivalent of VDUSE_GET_FEATURES on current Linux master. Or should the VDUSE instance assume that if the ioctl returns -EINVAL the kernel does not support them?

If we go that way, yes.

I'm ok with doing the change. Kind of, I still think that increasing the API version is way cleaner, so no ioctl returns any error in regular initialization. But python's style "ask forgiveness not permission" is also ok for me.

But I'm really missing the motivation of it. What's the problem with the increase of the version that you are trying to solve here?

Can we make a rule to know when to increase the version, and when it is not needed and it is ok for ioctls to just return -EINVAL or -ENOIOCTLCMD, so future series can hit the right behavior from earlier versions?

From your comment I extract that we don't need to increase the API version number if we just add ioctls commands to the device, like probing if an ioctl exists. Also, we don't need to increase the version if we can tell if a member in vduse_dev_config is known to exist because a previous ioctl (or feature flag) succeeded.

Am I missing something?

#8 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

Quoted 118 lines · Eugenio Perez Martin
Quoted 107 lines · Jason Wang
Quoted 104 lines · Eugenio Perez Martin
Quoted 101 lines · Jason Wang
Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Quoted 8 lines
Quoted 5 lines

How do you know if you can call VDUSE_GET_FEATURES and VDUSE_SET_FEATURES then? vduse_ioctl does not even return -ENOIOCMD but -EINVAL if you call the equivalent of VDUSE_GET_FEATURES on current Linux master. Or should the VDUSE instance assume that if the ioctl returns -EINVAL the kernel does not support them?

If we go that way, yes.

Quoted 7 lines

I'm ok with doing the change. Kind of, I still think that increasing the API version is way cleaner, so no ioctl returns any error in regular initialization. But python's style "ask forgiveness not permission" is also ok for me.

But I'm really missing the motivation of it. What's the problem with the increase of the version that you are trying to solve here?

For example, if a feature is simply added via a new ioctl. I think we don't need to bump the version. An example is the umem reg/dereg. VDUSE starts from this choice so I think we should stick to that.

I think a better time to bump the version are

1) an ioctl is expected to behave differently
2) new ioctls are mutually exclusive with one(s) of the existing ioctls
3) new initialization protocol

etc.

Quoted 5 lines

Can we make a rule to know when to increase the version, and when it is not needed and it is ok for ioctls to just return -EINVAL or -ENOIOCTLCMD, so future series can hit the right behavior from earlier versions?

See above.

Quoted 8 lines

From your comment I extract that we don't need to increase the API version number if we just add ioctls commands to the device, like probing if an ioctl exists. Also, we don't need to increase the version if we can tell if a member in vduse_dev_config is known to exist because a previous ioctl (or feature flag) succeeded.

Am I missing something?

I might be wrong, just some thoughts. Please correct me if I was wrong.

Thanks

Quoted 1 line
#9 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

Quoted 155 lines · Jason Wang
Quoted 118 lines · Eugenio Perez Martin
Quoted 107 lines · Jason Wang
Quoted 104 lines · Eugenio Perez Martin
Quoted 101 lines · Jason Wang
Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Quoted 8 lines
Quoted 5 lines

How do you know if you can call VDUSE_GET_FEATURES and VDUSE_SET_FEATURES then? vduse_ioctl does not even return -ENOIOCMD but -EINVAL if you call the equivalent of VDUSE_GET_FEATURES on current Linux master. Or should the VDUSE instance assume that if the ioctl returns -EINVAL the kernel does not support them?

If we go that way, yes.

Quoted 7 lines

I'm ok with doing the change. Kind of, I still think that increasing the API version is way cleaner, so no ioctl returns any error in regular initialization. But python's style "ask forgiveness not permission" is also ok for me.

But I'm really missing the motivation of it. What's the problem with the increase of the version that you are trying to solve here?

For example, if a feature is simply added via a new ioctl. I think we don't need to bump the version. An example is the umem reg/dereg. VDUSE starts from this choice so I think we should stick to that.

I think a better time to bump the version are

1) an ioctl is expected to behave differently
2) new ioctls are mutually exclusive with one(s) of the existing ioctls
3) new initialization protocol

etc.

Quoted 5 lines

Can we make a rule to know when to increase the version, and when it is not needed and it is ok for ioctls to just return -EINVAL or -ENOIOCTLCMD, so future series can hit the right behavior from earlier versions?

See above.

Quoted 7 lines

From your comment I extract that we don't need to increase the API version number if we just add ioctls commands to the device, like probing if an ioctl exists. Also, we don't need to increase the version if we can tell if a member in vduse_dev_config is known to exist because a previous ioctl (or feature flag) succeeded.

Quoted 7 lines
Quoted 1 line

Am I missing something?

I might be wrong, just some thoughts. Please correct me if I was wrong.

Thanks

Quoted 1 line

Not bumping version is nice as people can cherry pick features.

OTOH if people are cherry picking features then userspace has to support all kind of combinations of features.

So I think there can not be a simple rule, it boils down to this:
if userspace can handle all failures easily - just add an ioctl.

If userspace can not and would merely assert on success - update the version.

Signature · 1 line
MST
author maintainer
#10 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

In reply to Jason Wang
Quoted 126 lines · Jason Wang
Quoted 123 lines · Eugenio Perez Martin
Quoted 107 lines · Jason Wang
Quoted 104 lines · Eugenio Perez Martin
Quoted 101 lines · Jason Wang
Quoted 87 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 5 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Quoted 8 lines
Quoted 5 lines

How do you know if you can call VDUSE_GET_FEATURES and VDUSE_SET_FEATURES then? vduse_ioctl does not even return -ENOIOCMD but -EINVAL if you call the equivalent of VDUSE_GET_FEATURES on current Linux master. Or should the VDUSE instance assume that if the ioctl returns -EINVAL the kernel does not support them?

If we go that way, yes.

I'm ok with doing the change. Kind of, I still think that increasing the API version is way cleaner, so no ioctl returns any error in regular initialization. But python's style "ask forgiveness not permission" is also ok for me.

Quoted 7 lines
Quoted 2 lines

But I'm really missing the motivation of it. What's the problem with the increase of the version that you are trying to solve here?

For example, if a feature is simply added via a new ioctl. I think we don't need to bump the version. An example is the umem reg/dereg. VDUSE starts from this choice so I think we should stick to that.

That's a good example, yes.

Quoted 6 lines

I think a better time to bump the version are

1) an ioctl is expected to behave differently
2) new ioctls are mutually exclusive with one(s) of the existing ioctls
3) new initialization protocol

My line of thought was that the config struct initialization changed, but modern devices can also support no features so the features field keeps being 0.

Quoted 17 lines

etc.

Quoted 5 lines

Can we make a rule to know when to increase the version, and when it is not needed and it is ok for ioctls to just return -EINVAL or -ENOIOCTLCMD, so future series can hit the right behavior from earlier versions?

See above.

Quoted 7 lines

From your comment I extract that we don't need to increase the API version number if we just add ioctls commands to the device, like probing if an ioctl exists. Also, we don't need to increase the version if we can tell if a member in vduse_dev_config is known to exist because a previous ioctl (or feature flag) succeeded.

Quoted 4 lines
Quoted 1 line

Am I missing something?

I might be wrong, just some thoughts. Please correct me if I was wrong.

I'm happy with that, modern userland with the old kernel modules will just see -ENOIOCTLCMD and take the right path. I guess it's the same as umem reg/dereg.

Resending without bumping the API version.

author maintainer
#11 lore raw

Re: [PATCH v2 3/5] vduse: add VDUSE_GET_FEATURES ioctl

In reply to Jason Wang
Quoted 88 lines · Jason Wang
Quoted 86 lines · Eugenio Pérez

Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.

---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/vduse.h         |  7 ++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +28 -0
index d1da7c15d98b..7458cbaed4c7 100644
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -1947,6 +1950,19 @@ static bool vduse_validate_config(struct vduse_dev_config *config,
                         sizeof(config->reserved)))
                return false;
 
+       if (api_version < VDUSE_API_VERSION_2) {
+               if (config->vduse_features) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "config->vduse_features with version %llu",
+                               api_version);
+                       return false;
+               }
+       } else {
+               if (config->vduse_features & ~vduse_features)
+                       return false;
+       }
+
+
        if (api_version < VDUSE_API_VERSION_1 &&
            (config->ngroups || config->nas))
                return false;
@@ -2207,6 +2223,18 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
                ret = vduse_destroy_dev(name);
                break;
        }
+       case VDUSE_GET_FEATURES:
+               if (control->api_version < VDUSE_API_VERSION_2) {
+                       dev_dbg(vduse_ctrl_dev,
+                               "VDUSE_GET_FEATURES ioctl with version %llu",
+                               control->api_version);
+                       ret = -EINVAL;
+                       break;
+               }
+
+               ret = put_user(vduse_features, (u64 __user *)argp);
+               break;
+
        default:
                ret = -EINVAL;
                break;
include/uapi/linux/vduse.h +1 -0
index 27832d46084c..8898d9daa777 100644
@@ -37,6 +37,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space
Quoted 9 lines

include/uapi/linux/vduse.h struct vduse_dev_config { · line 57

Quoted 6 lines
include/uapi/linux/vduse.h +1 -1
@@ -53,7 +54,8 @@ struct vduse_dev_config {
        __u32 vq_align;
        __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
        __u32 nas; /* if VDUSE_API_VERSION >= 1 */
-       __u32 reserved[11];
+       __u64 vduse_features;

Nit: Should we document " /* if VDUSE_API_VERSION >= 2 */

(I forgot to reply to this one) sure, I'll add the comment for the next version.

Thanks!

Quoted 24 lines

But I have an open question, is this better to just introduce a VDUSE_SET_FEATURES then we can do negotiation to avoid bumping API versions?

Thanks

Quoted 17 lines
include/uapi/linux/vduse.h +4 -1
+       __u32 reserved[9];
        __u32 config_size;
        __u8 config[];
 };
@@ -67,6 +69,9 @@ struct vduse_dev_config {
  */
 #define VDUSE_DESTROY_DEV      _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
 
+/* Get the VDUSE supported features */
+#define VDUSE_GET_FEATURES     _IOR(VDUSE_BASE, 0x04, __u64)
+
 /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
 
 /**
--

2.53.0

author maintainer
#12 lore raw

PATCH v2 4/5 vduse: advertise API V2 support

In reply to Eugenio Pérez

Advertise VDUSE API V2 support to userspace.

With the necessary infrastructure and feature flags in place, VDUSE devices can now opt-in to the new API and utilize the VDUSE_F_QUEUE_READY feature.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +2 -2
index 7458cbaed4c7..39e115b8bf44 100644
@@ -2168,7 +2168,7 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
 	switch (cmd) {
 	case VDUSE_GET_API_VERSION:
 		if (control->api_version == VDUSE_API_VERSION_NOT_ASKED)
-			control->api_version = VDUSE_API_VERSION_1;
+			control->api_version = VDUSE_API_VERSION_2;
 		ret = put_user(control->api_version, (u64 __user *)argp);
 		break;
 	case VDUSE_SET_API_VERSION: {
@@ -2179,7 +2179,7 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
 			break;
 
 		ret = -EINVAL;
-		if (api_version > VDUSE_API_VERSION_1)
+		if (api_version > VDUSE_API_VERSION_2)
 			break;
 
 		ret = 0;
Signature · 1 line
2.53.0
author maintainer
#13 lore raw

PATCH v2 5/5 vduse: add F_QUEUE_READY feature

In reply to Eugenio Pérez

Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.

In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.

---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Set vq_ready before sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.

---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
 include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +27 -1
index 39e115b8bf44..59d9c4718d86 100644
@@ -9,6 +9,7 @@
  */
 
 #include "linux/virtio_net.h"
+#include <linux/bits.h>
 #include <linux/cleanup.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -53,7 +54,7 @@
 #define IRQ_UNBOUND -1
 
 /* Supported VDUSE features */
-static const uint64_t vduse_features;
+static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
 
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
@@ -120,6 +121,7 @@ struct vduse_dev {
 	char *name;
 	struct mutex lock;
 	spinlock_t msg_lock;
+	u64 vduse_features;
 	u64 msg_unique;
 	u32 msg_timeout;
 	wait_queue_head_t waitq;
@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
 {
 	struct vduse_dev *dev = vdpa_to_vduse(vdpa);
 	struct vduse_virtqueue *vq = dev->vqs[idx];
+	struct vduse_dev_msg msg = { 0 };
+	int r;
 
 	vq->ready = ready;
+
+	if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
+		return;
+
+	msg.req.type = VDUSE_SET_VQ_READY;
+	msg.req.vq_ready.num = idx;
+	msg.req.vq_ready.ready = !!ready;
+
+	r = vduse_dev_msg_sync(dev, &msg);
+
+	if (r < 0) {
+		dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
+			idx, ready);
+
+		/* We can't do better than break the device in this case */
+		spin_lock(&dev->msg_lock);
+		vduse_dev_broken(dev);
+		spin_unlock(&dev->msg_lock);
+	}
 }
 
 static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
@@ -2091,6 +2114,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	dev->device_features = config->features;
 	dev->device_id = config->device_id;
 	dev->vendor_id = config->vendor_id;
+	dev->vduse_features = config->vduse_features;
+	dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",
+		config->name, config->vduse_features);
 
 	dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
 	dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
include/uapi/linux/vduse.h +18 -0
index 8898d9daa777..d39734cef6d3 100644
@@ -18,6 +18,9 @@
 
 #define VDUSE_API_VERSION_2	2
 
+/* The VDUSE instance expects a request for vq ready */
+#define VDUSE_F_QUEUE_READY	0
+
 /*
  * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
  * This is used for future extension.
@@ -334,6 +337,7 @@ enum vduse_req_type {
 	VDUSE_SET_STATUS,
 	VDUSE_UPDATE_IOTLB,
 	VDUSE_SET_VQ_GROUP_ASID,
+	VDUSE_SET_VQ_READY,
 };
 
 /**
@@ -381,6 +385,15 @@ struct vduse_iova_range_v2 {
 	__u32 padding;
 };
 
+/**
+ * struct vduse_vq_ready - Virtqueue ready request message
+ * @num: Virtqueue number
+ */
+struct vduse_vq_ready {
+	__u32 num;
+	__u32 ready;
+};
+
 /**
  * struct vduse_dev_request - control request
  * @type: request type
@@ -391,6 +404,7 @@ struct vduse_iova_range_v2 {
  * @iova: IOVA range for updating
  * @iova_v2: IOVA range for updating if API_VERSION >= 1
  * @vq_group_asid: ASID of a virtqueue group
+ * @vq_ready: Virtqueue ready request
  * @padding: padding
  *
  * Structure used by read(2) on /dev/vduse/$NAME.
@@ -408,6 +422,10 @@ struct vduse_dev_request {
 		 */
 		struct vduse_iova_range_v2 iova_v2;
 		struct vduse_vq_group_asid vq_group_asid;
+
+		/* Only if VDUSE_F_QUEUE_READY is negotiated */
+		struct vduse_vq_ready vq_ready;
+
 		__u32 padding[32];
 	};
 };
Signature · 1 line
2.53.0