Add queue ready message to VDUSE
Eugenio Pérez
maintainerv3 cover Add queue ready message to VDUSE
This series introduces a new VDUSE message for VDUSE userland instance to detect when a VirtQueue (VQ) is enabled, replacing the polling.
VirtIO net devices' dataplane is started after the control virtqueue so QEMU can apply the configuration in the destination of a Live Migration. Without this feature, the VDUSE instance must poll the VQs to check when (and if) a VQ has been enabled.
This series also implements VDUSE feature flags allowing the VDUSE devices to opt-in to the VQ ready message. Devices that opt-in to this feature will receive explicit notifications when a VQ is ready. Devices that do not set this flag remain unaffected, ensuring backward compatibility without indefinitely incrementing API versions.
The VDUSE features is a 64 bit bitmap for simplicity, the same way as vhost and vhost-net started. It can be extended as a flexible array of bits when we reach so many features, but it seems unlikely at this point.
Error cases tested:
* Call VDUSE_GET_FEATURES without get the API VERSION (so API == 0) and with API VERSION set to 1 with VDUSE_SET_API_VERSION (-EINVAL returned from VDUSE_GET_FEATURES ioctl).
* Try to create a device with config->features different than 0 without fetch the api version (so API == 0) and with API_VERSION set to 1 (-EINVAL returned from VDUSE_CREATE_DEV ioctl).
* Test regular initialization of single queue devices with V2 with and without VDUSE_F_QUEUE_READY set in device->config.
* Test expected behavior when VDUSE userland instance returns VDUSE_REQ_RESULT_FAILED from VDUSE_SET_VQ_READY message.
* Repeat all the tests with multiqueue devices, by reverting 56e71885b0349 ("vduse: Temporarily fail if control queue feature requested").
This series depends on
https://lore.kernel.org/lkml/[email protected]/
v3:
* Remove API_VERSION bump to 2
* Add comment about struct vduse_dev_config:vduse_features is only valid if VDUSE_GET_FEATURES success.
v2:
* Fix comment of vduse_dev_request.vq_ready
* Set vq_ready before sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.
* Return -EINVAL if control ioctl called with version < 2, so userland visible reply is kept (Jason).
Eugenio Pérez (3):
vduse: store control device pointer
vduse: add VDUSE_GET_FEATURES ioctl
vduse: add F_QUEUE_READY feature
drivers/vdpa/vdpa_user/vduse_dev.c | 41 +++++++++++++++++++++++++++--- include/uapi/linux/vduse.h | 25 +++++++++++++++++- 2 files changed, 61 insertions(+), 5 deletions(-)
Signature · 1 line
2.53.0
Eugenio Pérez
maintainerPATCH v3 1/3 vduse: store control device pointer
This helps log the errors in next patches. The alternative is to perform a linear search for it with class_find_device_by_devt(class, devt), as device_destroy do for cleaning.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/vdpa/vdpa_user/vduse_dev.c +4 -4
@@ -164,6 +164,7 @@ static DEFINE_IDR(vduse_idr);static dev_t vduse_major;static struct cdev vduse_ctrl_cdev; +static const struct device *vduse_ctrl_dev;static struct cdev vduse_cdev;static struct workqueue_struct *vduse_irq_wq;static struct workqueue_struct *vduse_irq_bound_wq;
@@ -2396,7 +2397,6 @@ static void vduse_mgmtdev_exit(void)static int vduse_init (void ) {int ret; -struct device *dev; ret =class_register (&vduse_class);if (ret)
@@ -2413,9 +2413,9 @@ static int vduse_init(void)if (ret) goto err_ctrl_cdev; - dev =device_create (&vduse_class,NULL , vduse_major,NULL ,"control" ); -if (IS_ERR (dev)) { - ret =PTR_ERR (dev); + vduse_ctrl_dev =device_create (&vduse_class,NULL , vduse_major,NULL ,"control" ); +if (IS_ERR (vduse_ctrl_dev)) { + ret =PTR_ERR (vduse_ctrl_dev); goto err_device; }
Signature · 1 line
2.53.0
Eugenio Pérez
maintainerPATCH v3 2/3 vduse: add VDUSE_GET_FEATURES ioctl
In reply to Eugenio PérezAdd an ioctl to allow VDUSE instances to query the available features supported by the kernel module.
---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.
Also, dev_dbg is used for logging rather than dev_err as these can be triggered from userspace.
---
v3:
* Remove check for API_VERSION < 2
* Add comment about struct vduse_dev_config:vduse_features is only valid if VDUSE_GET_FEATURES success.
v2:
* return -EINVAL if ioctl called with version < 2, so userland visible reply is kept (Jason).
--- drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++ include/uapi/linux/vduse.h | 7 ++++++- 2 files changed, 13 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +7 -0
@@ -52,6 +52,9 @@#define IRQ_UNBOUND -1 +/* Supported VDUSE features */ +static const uint64_t vduse_features; +/* * VDUSE instance have not asked the vduse API version, so assume 0. *
@@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd, ret =vduse_destroy_dev (name);break ; } +case VDUSE_GET_FEATURES : + ret =put_user (vduse_features, (u64 __user *)argp); +break ; +default : ret = -EINVAL ;break ;
include/uapi/linux/vduse.h +6 -1
@@ -33,6 +33,7 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares +* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space
@@ -49,7 +50,8 @@ struct vduse_dev_config {__u32 vq_align;__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u32 reserved[11 ]; +__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ +__u32 reserved[9 ];__u32 config_size;__u8 config[]; };
@@ -63,6 +65,9 @@ struct vduse_dev_config {*/ #define VDUSE_DESTROY_DEV _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX]) +/* Get the VDUSE supported features */ +#define VDUSE_GET_FEATURES _IOR(VDUSE_BASE, 0x04, __u64) +/* The ioctls for VDUSE device (/dev/vduse/$NAME) */ /**
Signature · 1 line
2.53.0
Re: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl
Quoted 64 lines · Eugenio Pérez
Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.
Signed-off-by Eugenio Pérez <[email protected]>---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.Also,
dev_dbgis used for logging rather thandev_erras these can be triggered from userspace.
---
v3:
* Remove check forAPI_VERSION< 2
* Add comment about struct vduse_dev_config:vduse_features is only valid ifVDUSE_GET_FEATURESsuccess.v2:
* return-EINVALif ioctl called with version < 2, so userland visible reply is kept (Jason).--- drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++ include/uapi/linux/vduse.h | 7 ++++++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +7
-0@@ -52,6 +52,9 @@#define IRQ_UNBOUND -1 +/* Supported VDUSE features */ +static const uint64_t vduse_features; +/* * VDUSE instance have not asked the vduse API version, so assume 0. * @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd, ret =vduse_destroy_dev (name);break ; } +case VDUSE_GET_FEATURES : + ret =put_user (vduse_features, (u64 __user *)argp); +break ; +default : ret = -EINVAL ;break ;include/uapi/linux/vduse.h +1
-0@@ -33,6 +33,7 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares +* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space
include/uapi/linux/vduse.h struct vduse_dev_config { · line 54
Quoted 6 lines
include/uapi/linux/vduse.h +2
-1@@ -49,7 +50,8 @@ struct vduse_dev_config {__u32 vq_align;__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u32 reserved[11 ]; +__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ +__u32 reserved[9 ];
If we go with VDUSE_GET_FEATURES, I'd perfer to go with VDUSE_SET_FEATURES intead of this.
But we can hear from others.
Quoted 14 lines
include/uapi/linux/vduse.h +3
-1__u32 config_size;__u8 config[]; };@@ -63,6 +65,9 @@ struct vduse_dev_config {*/ #define VDUSE_DESTROY_DEV _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX]) +/* Get the VDUSE supported features */ +#define VDUSE_GET_FEATURES _IOR(VDUSE_BASE, 0x04, __u64) +/* The ioctls for VDUSE device (/dev/vduse/$NAME) */ /** --
Quoted 2 lines
2.53.0
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl
Quoted 67 lines · Jason Wang
Quoted 65 lines · Eugenio Pérez
Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.
Signed-off-by Eugenio Pérez <[email protected]>---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.Also,
dev_dbgis used for logging rather thandev_erras these can be triggered from userspace.
---
v3:
* Remove check forAPI_VERSION< 2
* Add comment about struct vduse_dev_config:vduse_features is only valid ifVDUSE_GET_FEATURESsuccess.v2:
* return-EINVALif ioctl called with version < 2, so userland visible reply is kept (Jason).--- drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++ include/uapi/linux/vduse.h | 7 ++++++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +7
-0@@ -52,6 +52,9 @@#define IRQ_UNBOUND -1 +/* Supported VDUSE features */ +static const uint64_t vduse_features; +/* * VDUSE instance have not asked the vduse API version, so assume 0. * @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd, ret =vduse_destroy_dev (name);break ; } +case VDUSE_GET_FEATURES : + ret =put_user (vduse_features, (u64 __user *)argp); +break ; +default : ret = -EINVAL ;break ;include/uapi/linux/vduse.h +1
-0@@ -33,6 +33,7 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares +* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space @@ -49,7 +50,8 @@ struct vduse_dev_config {__u32 vq_align;
Quoted 9 lines
include/uapi/linux/vduse.h struct vduse_dev_config { · line 54
Quoted 5 lines
include/uapi/linux/vduse.h +2
-1__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u32 reserved[11 ]; +__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ +__u32 reserved[9 ];If we go with
VDUSE_GET_FEATURES, I'd perfer to go withVDUSE_SET_FEATURESintead of this.
I didn't realize the lack of symmetry, but that approach grows and complicates the possible states of struct vduse_control and the vduse_ioctl code for little benefit in my opinion. Making it atomic in VDUSE_CREATE_DEV ioctl helps centralize all the potential errors. Also, from previous experience with vhost_vdpa, the ioctls slow down the device creation, which could affect things like VDUSE adoption for containers etc.
On the other hand, it would make it easier to tell if the device couldn't be created because of invalid features set. This is highly unlikely, as the VDUSE device should retrieve the features by calling VDUSE_GET_FEATURES earlier.
I don't think it is worth it, but if you think we should go that way I'm ok with the change.
Quoted 21 lines
But we can hear from others.
Quoted 14 lines
include/uapi/linux/vduse.h +3
-1__u32 config_size;__u8 config[]; };@@ -63,6 +65,9 @@ struct vduse_dev_config {*/ #define VDUSE_DESTROY_DEV _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX]) +/* Get the VDUSE supported features */ +#define VDUSE_GET_FEATURES _IOR(VDUSE_BASE, 0x04, __u64) +/* The ioctls for VDUSE device (/dev/vduse/$NAME) */ /** -- Quoted 2 lines
2.53.0
Thanks
Re: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl
Quoted 79 lines · Eugenio Perez Martin
Quoted 67 lines · Jason Wang
Quoted 65 lines · Eugenio Pérez
Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.
Signed-off-by Eugenio Pérez <[email protected]>---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.Also,
dev_dbgis used for logging rather thandev_erras these can be triggered from userspace.
---
v3:
* Remove check forAPI_VERSION< 2
* Add comment about struct vduse_dev_config:vduse_features is only valid ifVDUSE_GET_FEATURESsuccess.v2:
* return-EINVALif ioctl called with version < 2, so userland visible reply is kept (Jason).--- drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++ include/uapi/linux/vduse.h | 7 ++++++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +7
-0@@ -52,6 +52,9 @@#define IRQ_UNBOUND -1 +/* Supported VDUSE features */ +static const uint64_t vduse_features; +/* * VDUSE instance have not asked the vduse API version, so assume 0. * @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd, ret =vduse_destroy_dev (name);break ; } +case VDUSE_GET_FEATURES : + ret =put_user (vduse_features, (u64 __user *)argp); +break ; +default : ret = -EINVAL ;break ;include/uapi/linux/vduse.h +1
-0@@ -33,6 +33,7 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares +* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space @@ -49,7 +50,8 @@ struct vduse_dev_config {__u32 vq_align;Quoted 9 lines
include/uapi/linux/vduse.h struct vduse_dev_config { · line 54
Quoted 5 lines
include/uapi/linux/vduse.h +2
-1__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u32 reserved[11 ]; +__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ +__u32 reserved[9 ];If we go with
VDUSE_GET_FEATURES, I'd perfer to go withVDUSE_SET_FEATURESintead of this.
Quoted 7 lines
I didn't realize the lack of symmetry, but that approach grows and complicates the possible states of struct
vduse_controland thevduse_ioctlcode for little benefit in my opinion. Making it atomic inVDUSE_CREATE_DEVioctl helps centralize all the potential errors. Also, from previous experience withvhost_vdpa, the ioctls slow down the device creation, which could affect things like VDUSE adoption for containers etc.
Maybe, but if we really care about the time spent on device creation, uAPI needs redesigning. Or at least more feature in the future wouldn't slow down further.
Quoted 8 lines
On the other hand, it would make it easier to tell if the device couldn't be created because of invalid features set. This is highly unlikely, as the VDUSE device should retrieve the features by calling
VDUSE_GET_FEATURESearlier.I don't think it is worth it, but if you think we should go that way I'm ok with the change.
It depends on the intialziation flow. After getting features, I'd expect it's something like this
A:
1) GET_API_VERSION
2) SET_API_VERSION
3) GET_FEATURES
4) SET_FEATURES
5) CREATE_DEV
Or B in this patch:
1) GET_API_VERSION
2) SET_API_VERSION
3) GET_FEATURES
4) CREATE_DEV
I think it's better to stick the symmetry as API_VERSION (technically, API could be set via config as well?)
BTW, I found VDUSE_SET_API_VERSION could be set multiple times (even after the device is create), do we need to fix this?
Thanks
Quoted 23 lines
Quoted 22 lines
But we can hear from others.
Quoted 14 lines
include/uapi/linux/vduse.h +3
-1__u32 config_size;__u8 config[]; };@@ -63,6 +65,9 @@ struct vduse_dev_config {*/ #define VDUSE_DESTROY_DEV _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX]) +/* Get the VDUSE supported features */ +#define VDUSE_GET_FEATURES _IOR(VDUSE_BASE, 0x04, __u64) +/* The ioctls for VDUSE device (/dev/vduse/$NAME) */ /** -- Quoted 2 lines
2.53.0
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl
Quoted 116 lines · Jason Wang
Quoted 79 lines · Eugenio Perez Martin
Quoted 67 lines · Jason Wang
Quoted 65 lines · Eugenio Pérez
Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.
Signed-off-by Eugenio Pérez <[email protected]>---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.Also,
dev_dbgis used for logging rather thandev_erras these can be triggered from userspace.
---
v3:
* Remove check forAPI_VERSION< 2
* Add comment about struct vduse_dev_config:vduse_features is only valid ifVDUSE_GET_FEATURESsuccess.v2:
* return-EINVALif ioctl called with version < 2, so userland visible reply is kept (Jason).--- drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++ include/uapi/linux/vduse.h | 7 ++++++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +7
-0@@ -52,6 +52,9 @@#define IRQ_UNBOUND -1 +/* Supported VDUSE features */ +static const uint64_t vduse_features; +/* * VDUSE instance have not asked the vduse API version, so assume 0. * @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd, ret =vduse_destroy_dev (name);break ; } +case VDUSE_GET_FEATURES : + ret =put_user (vduse_features, (u64 __user *)argp); +break ; +default : ret = -EINVAL ;break ;include/uapi/linux/vduse.h +1
-0@@ -33,6 +33,7 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares +* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space @@ -49,7 +50,8 @@ struct vduse_dev_config {__u32 vq_align;Quoted 9 lines
include/uapi/linux/vduse.h struct vduse_dev_config { · line 54
Quoted 5 lines
include/uapi/linux/vduse.h +2
-1__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u32 reserved[11 ]; +__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ +__u32 reserved[9 ];If we go with
VDUSE_GET_FEATURES, I'd perfer to go withVDUSE_SET_FEATURESintead of this.Quoted 7 lines
I didn't realize the lack of symmetry, but that approach grows and complicates the possible states of struct
vduse_controland thevduse_ioctlcode for little benefit in my opinion. Making it atomic inVDUSE_CREATE_DEVioctl helps centralize all the potential errors. Also, from previous experience withvhost_vdpa, the ioctls slow down the device creation, which could affect things like VDUSE adoption for containers etc.Maybe, but if we really care about the time spent on device creation, uAPI needs redesigning. Or at least more feature in the future wouldn't slow down further.
Quoted 8 lines
On the other hand, it would make it easier to tell if the device couldn't be created because of invalid features set. This is highly unlikely, as the VDUSE device should retrieve the features by calling
VDUSE_GET_FEATURESearlier.I don't think it is worth it, but if you think we should go that way I'm ok with the change.
It depends on the intialziation flow. After getting features, I'd expect it's something like this
A:
1)
GET_API_VERSION
2)SET_API_VERSION
3)GET_FEATURES
4)SET_FEATURES
5)CREATE_DEVOr B in this patch:
Quoted 8 lines
1)
GET_API_VERSION
2)SET_API_VERSION
3)GET_FEATURES
4)CREATE_DEVI think it's better to stick the symmetry as
API_VERSION(technically, API could be set via config as well?)
Yes, I'd have chosen to set the API_VERSION via config too.
Quoted 3 lines
BTW, I found
VDUSE_SET_API_VERSIONcould be set multiple times (even after the device is create), do we need to fix this?
It shouldn't be a problem, each device has its own copy of api_version. It seems clear to me that the API setting only affects devices created after the change, but I can add a guard after the first device is created.
Quoted 26 lines
Thanks
Quoted 23 lines
Quoted 22 lines
But we can hear from others.
Quoted 14 lines
include/uapi/linux/vduse.h +3
-1__u32 config_size;__u8 config[]; };@@ -63,6 +65,9 @@ struct vduse_dev_config {*/ #define VDUSE_DESTROY_DEV _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX]) +/* Get the VDUSE supported features */ +#define VDUSE_GET_FEATURES _IOR(VDUSE_BASE, 0x04, __u64) +/* The ioctls for VDUSE device (/dev/vduse/$NAME) */ /** -- Quoted 2 lines
2.53.0
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl
Quoted 129 lines · Eugenio Perez Martin
Quoted 116 lines · Jason Wang
Quoted 79 lines · Eugenio Perez Martin
Quoted 67 lines · Jason Wang
Quoted 65 lines · Eugenio Pérez
Add an ioctl to allow VDUSE instances to query the available features supported by the kernel module.
Signed-off-by Eugenio Pérez <[email protected]>---
A simple u64 bitmap is used for feature flags. While a flexible array could support indefinite expansion, 64 bits is sufficient for the foreseeable future and simplifies the implementation.Also,
dev_dbgis used for logging rather thandev_erras these can be triggered from userspace.
---
v3:
* Remove check forAPI_VERSION< 2
* Add comment about struct vduse_dev_config:vduse_features is only valid ifVDUSE_GET_FEATURESsuccess.v2:
* return-EINVALif ioctl called with version < 2, so userland visible reply is kept (Jason).--- drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++ include/uapi/linux/vduse.h | 7 ++++++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +7
-0@@ -52,6 +52,9 @@#define IRQ_UNBOUND -1 +/* Supported VDUSE features */ +static const uint64_t vduse_features; +/* * VDUSE instance have not asked the vduse API version, so assume 0. * @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd, ret =vduse_destroy_dev (name);break ; } +case VDUSE_GET_FEATURES : + ret =put_user (vduse_features, (u64 __user *)argp); +break ; +default : ret = -EINVAL ;break ;include/uapi/linux/vduse.h +1
-0@@ -33,6 +33,7 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares +* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space @@ -49,7 +50,8 @@ struct vduse_dev_config {__u32 vq_align;Quoted 9 lines
include/uapi/linux/vduse.h struct vduse_dev_config { · line 54
Quoted 5 lines
include/uapi/linux/vduse.h +2
-1__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u32 reserved[11 ]; +__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ +__u32 reserved[9 ];If we go with
VDUSE_GET_FEATURES, I'd perfer to go withVDUSE_SET_FEATURESintead of this.Quoted 7 lines
I didn't realize the lack of symmetry, but that approach grows and complicates the possible states of struct
vduse_controland thevduse_ioctlcode for little benefit in my opinion. Making it atomic inVDUSE_CREATE_DEVioctl helps centralize all the potential errors. Also, from previous experience withvhost_vdpa, the ioctls slow down the device creation, which could affect things like VDUSE adoption for containers etc.Maybe, but if we really care about the time spent on device creation, uAPI needs redesigning. Or at least more feature in the future wouldn't slow down further.
Quoted 8 lines
On the other hand, it would make it easier to tell if the device couldn't be created because of invalid features set. This is highly unlikely, as the VDUSE device should retrieve the features by calling
VDUSE_GET_FEATURESearlier.I don't think it is worth it, but if you think we should go that way I'm ok with the change.
It depends on the intialziation flow. After getting features, I'd expect it's something like this
A:
1)
GET_API_VERSION
2)SET_API_VERSION
3)GET_FEATURES
4)SET_FEATURES
5)CREATE_DEVOr B in this patch:
Quoted 8 lines
1)
GET_API_VERSION
2)SET_API_VERSION
3)GET_FEATURES
4)CREATE_DEVI think it's better to stick the symmetry as
API_VERSION(technically, API could be set via config as well?)Yes, I'd have chosen to set the
API_VERSIONvia config too.
Quoted 9 lines
Quoted 3 lines
BTW, I found
VDUSE_SET_API_VERSIONcould be set multiple times (even after the device is create), do we need to fix this?It shouldn't be a problem, each device has its own copy of
api_version. It seems clear to me that the API setting only affects devices created after the change, but I can add a guard after the first device is created.
I keep thinking that we are increasing the complexity for little benefit TBH. This is the diff on top of the current series. MST, do you prefer a new VDUSE_SET_FEATURES ioctl to follow the api version API and keep the symmetry with GET_VDUSE_FEATURES or use the config struct like the group and asid configuration?
--- drivers/vdpa/vdpa_user/vduse_dev.c | 27 +++++++++++++++++++++++---- include/uapi/linux/vduse.h | 7 ++++--- 2 files changed, 27 insertions(+), 7 deletions(-)
(quoted diff) +0 -0
b/drivers/vdpa/vdpa_user/vduse_dev.c
index 4f642b95a7cb..a5427dcebf29 100644
drivers/vdpa/vdpa_user/vduse_dev.c +3 -1
@@ -162,6 +162,7 @@ struct vduse_dev_msg {struct vduse_control {u64 api_version; +u64 vduse_features; };static DEFINE_MUTEX (vduse_lock);
@@ -2079,7 +2080,8 @@ static struct attribute *vduse_dev_attrs[] = {ATTRIBUTE_GROUPS (vduse_dev);static int vduse_create_dev (struct vduse_dev_config *config, -void *config_buf,u64 api_version) +void *config_buf,u64 api_version, +u64 vduse_features) {int ret;struct vduse_dev *dev;
@@ -2101,9 +2103,9 @@ static int vduse_create_dev(struct
vduse_dev_config *config,
dev->device_features = config->features;
dev->device_id = config->device_id;
dev->vendor_id = config->vendor_id;
- dev->vduse_features = config->vduse_features;
+ dev->vduse_features = vduse_features;
dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",- config->name, config->vduse_features);
+ config->name, vduse_features);
dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
@@ -2199,6 +2201,21 @@ static long vduse_ioctl(struct file *file,
unsigned int cmd,
control->api_version = api_version;
break;
}+ case VDUSE_SET_FEATURES: {
+ u64 features;
+
+ ret = -EFAULT;
+ if (get_user(features, (u64 __user *)argp))
+ break;
+
+ ret = -EINVAL;
+ if (features & ~vduse_features)
+ break;
+
+ ret = 0;
+ control->vduse_features = features;
+ break;
+ }
case VDUSE_CREATE_DEV: {
struct vduse_dev_config config;
unsigned long size = offsetof(struct vduse_dev_config, config);
@@ -2220,7 +2237,8 @@ static long vduse_ioctl(struct file *file,
unsigned int cmd,
break;
}
config.name[VDUSE_NAME_MAX - 1] = '\0';
- ret = vduse_create_dev(&config, buf, control->api_version);
+ ret = vduse_create_dev(&config, buf, control->api_version,
+ control->vduse_features);
if (ret)
kvfree(buf);
break;
@@ -2266,6 +2284,7 @@ static int vduse_open(struct inode *inode,
struct file *file)
return -ENOMEM;
control->api_version = VDUSE_API_VERSION_NOT_ASKED;
+ control->vduse_features = 0;
file->private_data = control;
return 0;include/uapi/linux/vduse.h +4 -3
@@ -26,6 +26,9 @@/* Set the version of VDUSE API that userspace supported. */ #define VDUSE_SET_API_VERSION _IOW(VDUSE_BASE, 0x01, __u64) +/* Set the VDUSE features that userspace supported. */ +#define VDUSE_SET_FEATURES _IOW(VDUSE_BASE, 0x05, __u64) +/** * struct vduse_dev_config - basic configuration of a VDUSE device * @name: VDUSE device name, needs to be NUL terminated
@@ -36,7 +39,6 @@* @vq_align: the allocation alignment of virtqueue's metadata * @ngroups: number of vq groups that VDUSE device declares * @nas: number of address spaces that VDUSE device declares -* @vduse_features: VDUSE features * @reserved: for future use, needs to be initialized to zero * @config_size: the size of the configuration space * @config: the buffer of the configuration space
@@ -53,8 +55,7 @@ struct vduse_dev_config {__u32 vq_align;__u32 ngroups;/* if VDUSE_API_VERSION >= 1 */ __u32 nas;/* if VDUSE_API_VERSION >= 1 */ -__u64 vduse_features;/* if VDUSE_GET_FEATURES is not EINVAL */ -__u32 reserved[9 ]; +__u32 reserved[11 ];__u32 config_size;__u8 config[]; };
Eugenio Pérez
maintainerPATCH v3 3/3 vduse: add F_QUEUE_READY feature
In reply to Eugenio PérezAdd the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.
In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Set vq_ready before sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.
--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_user/vduse_dev.c +27 -1
@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h>
@@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0.
@@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;
@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ; + msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ +spin_lock (&dev->msg_lock); +vduse_dev_broken (dev); +spin_unlock (&dev->msg_lock); + } }static bool vduse_vdpa_get_vq_ready (struct vdpa_device *vdpa,u16 idx)
@@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->device_features = config->features; dev->device_id = config->device_id; dev->vendor_id = config->vendor_id; + dev->vduse_features = config->vduse_features; +dev_dbg (vduse_ctrl_dev,"Creating device %s with features 0x%llx" , + config->name, config->vduse_features); dev->nas = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->nas; dev->as =kcalloc (dev->nas,sizeof (dev->as[0 ]),GFP_KERNEL );
include/uapi/linux/vduse.h +18 -0
@@ -14,6 +14,9 @@#define VDUSE_API_VERSION_1 1 +/* The VDUSE instance expects a request for vq ready */ +#define VDUSE_F_QUEUE_READY 0 +/* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension.
@@ -330,6 +333,7 @@ enum vduse_req_type {VDUSE_SET_STATUS ,VDUSE_UPDATE_IOTLB ,VDUSE_SET_VQ_GROUP_ASID , + VDUSE_SET_VQ_READY, };/**
@@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {__u32 padding; }; +/** +* struct vduse_vq_ready - Virtqueue ready request message +* @num: Virtqueue number +*/ +struct vduse_vq_ready { +__u32 num; +__u32 ready; +}; +/** * struct vduse_dev_request - control request * @type: request type
@@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {* @iova: IOVA range for updating * @iova_v2: IOVA range for updating if API_VERSION >= 1 * @vq_group_asid: ASID of a virtqueue group +* @vq_ready: Virtqueue ready request * @padding: padding * * Structure used by read(2) on /dev/vduse/$NAME.
@@ -404,6 +418,10 @@ struct vduse_dev_request {*/ struct vduse_iova_range_v2 iova_v2;struct vduse_vq_group_asid vq_group_asid; + +/* Only if VDUSE_F_QUEUE_READY is negotiated */ +struct vduse_vq_ready vq_ready; +__u32 padding[32 ]; }; };
Signature · 1 line
2.53.0
Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;
drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */
It's better to explain why we can't depend on vduse_dev_msg_sync() here.
For example it did:
if (unlikely(dev->broken))
return -EIO;
init_waitqueue_head(&msg->waitq);
spin_lock(&dev->msg_lock);
if (unlikely(dev->broken)) {
spin_unlock(&dev->msg_lock);
return -EIO;
}and
if (!msg->completed) {
list_del(&msg->list);
msg->resp.result = VDUSE_REQ_RESULT_FAILED;
/* Mark the device as malfunction when there is a timeout */
if (!ret)
vduse_dev_broken(dev);
}Quoted 75 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-0+spin_lock (&dev->msg_lock); +vduse_dev_broken (dev); +spin_unlock (&dev->msg_lock); + } }static bool vduse_vdpa_get_vq_ready (struct vdpa_device *vdpa,u16 idx)@@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->device_features = config->features; dev->device_id = config->device_id; dev->vendor_id = config->vendor_id; + dev->vduse_features = config->vduse_features; +dev_dbg (vduse_ctrl_dev,"Creating device %s with features 0x%llx" , + config->name, config->vduse_features); dev->nas = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->nas; dev->as =kcalloc (dev->nas,sizeof (dev->as[0 ]),GFP_KERNEL );include/uapi/linux/vduse.h +18
-1@@ -14,6 +14,9 @@#define VDUSE_API_VERSION_1 1 +/* The VDUSE instance expects a request for vq ready */ +#define VDUSE_F_QUEUE_READY 0 +/* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension. @@ -330,6 +333,7 @@ enum vduse_req_type {VDUSE_SET_STATUS ,VDUSE_UPDATE_IOTLB ,VDUSE_SET_VQ_GROUP_ASID , + VDUSE_SET_VQ_READY, };/** @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {__u32 padding; }; +/** +* struct vduse_vq_ready - Virtqueue ready request message +* @num: Virtqueue number +*/ +struct vduse_vq_ready { +__u32 num; +__u32 ready; +}; +/** * struct vduse_dev_request - control request * @type: request type @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {* @iova: IOVA range for updating * @iova_v2: IOVA range for updating if API_VERSION >= 1 * @vq_group_asid: ASID of a virtqueue group +* @vq_ready: Virtqueue ready request * @padding: padding * * Structure used by read(2) on /dev/vduse/$NAME. @@ -404,6 +418,10 @@ struct vduse_dev_request {*/ struct vduse_iova_range_v2 iova_v2;struct vduse_vq_group_asid vq_group_asid; + +/* Only if VDUSE_F_QUEUE_READY is negotiated */ +struct vduse_vq_ready vq_ready; +__u32 padding[32 ]; }; }; --
Quoted 2 lines
2.53.0
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) {
list_del(&msg->list);
msg->resp.result = VDUSE_REQ_RESULT_FAILED;
/* Mark the device as malfunction when there is a timeout */
if (!ret)
vduse_dev_broken(dev);
}I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on vduse_dev_msg_sync to call vduse_dev_broken(dev) by itself?
Quoted 80 lines
Quoted 75 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-0+spin_lock (&dev->msg_lock); +vduse_dev_broken (dev); +spin_unlock (&dev->msg_lock); + } }static bool vduse_vdpa_get_vq_ready (struct vdpa_device *vdpa,u16 idx)@@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->device_features = config->features; dev->device_id = config->device_id; dev->vendor_id = config->vendor_id; + dev->vduse_features = config->vduse_features; +dev_dbg (vduse_ctrl_dev,"Creating device %s with features 0x%llx" , + config->name, config->vduse_features); dev->nas = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->nas; dev->as =kcalloc (dev->nas,sizeof (dev->as[0 ]),GFP_KERNEL );include/uapi/linux/vduse.h +18
-1@@ -14,6 +14,9 @@#define VDUSE_API_VERSION_1 1 +/* The VDUSE instance expects a request for vq ready */ +#define VDUSE_F_QUEUE_READY 0 +/* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension. @@ -330,6 +333,7 @@ enum vduse_req_type {VDUSE_SET_STATUS ,VDUSE_UPDATE_IOTLB ,VDUSE_SET_VQ_GROUP_ASID , + VDUSE_SET_VQ_READY, };/** @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {__u32 padding; }; +/** +* struct vduse_vq_ready - Virtqueue ready request message +* @num: Virtqueue number +*/ +struct vduse_vq_ready { +__u32 num; +__u32 ready; +}; +/** * struct vduse_dev_request - control request * @type: request type @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {* @iova: IOVA range for updating * @iova_v2: IOVA range for updating if API_VERSION >= 1 * @vq_group_asid: ASID of a virtqueue group +* @vq_ready: Virtqueue ready request * @padding: padding * * Structure used by read(2) on /dev/vduse/$NAME. @@ -404,6 +418,10 @@ struct vduse_dev_request {*/ struct vduse_iova_range_v2 iova_v2;struct vduse_vq_group_asid vq_group_asid; + +/* Only if VDUSE_F_QUEUE_READY is negotiated */ +struct vduse_vq_ready vq_ready; +__u32 padding[32 ]; }; }; --Quoted 2 lines
2.53.0
Thanks
Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }
Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Thanks
Quoted 82 lines
Quoted 81 lines
Quoted 76 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-0+spin_lock (&dev->msg_lock); +vduse_dev_broken (dev); +spin_unlock (&dev->msg_lock); + } }static bool vduse_vdpa_get_vq_ready (struct vdpa_device *vdpa,u16 idx)@@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->device_features = config->features; dev->device_id = config->device_id; dev->vendor_id = config->vendor_id; + dev->vduse_features = config->vduse_features; +dev_dbg (vduse_ctrl_dev,"Creating device %s with features 0x%llx" , + config->name, config->vduse_features); dev->nas = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->nas; dev->as =kcalloc (dev->nas,sizeof (dev->as[0 ]),GFP_KERNEL );include/uapi/linux/vduse.h +18
-1@@ -14,6 +14,9 @@#define VDUSE_API_VERSION_1 1 +/* The VDUSE instance expects a request for vq ready */ +#define VDUSE_F_QUEUE_READY 0 +/* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension. @@ -330,6 +333,7 @@ enum vduse_req_type {VDUSE_SET_STATUS ,VDUSE_UPDATE_IOTLB ,VDUSE_SET_VQ_GROUP_ASID , + VDUSE_SET_VQ_READY, };/** @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {__u32 padding; }; +/** +* struct vduse_vq_ready - Virtqueue ready request message +* @num: Virtqueue number +*/ +struct vduse_vq_ready { +__u32 num; +__u32 ready; +}; +/** * struct vduse_dev_request - control request * @type: request type @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {* @iova: IOVA range for updating * @iova_v2: IOVA range for updating if API_VERSION >= 1 * @vq_group_asid: ASID of a virtqueue group +* @vq_ready: Virtqueue ready request * @padding: padding * * Structure used by read(2) on /dev/vduse/$NAME. @@ -404,6 +418,10 @@ struct vduse_dev_request {*/ struct vduse_iova_range_v2 iova_v2;struct vduse_vq_group_asid vq_group_asid; + +/* Only if VDUSE_F_QUEUE_READY is negotiated */ +struct vduse_vq_ready vq_ready; +__u32 padding[32 ]; }; }; --Quoted 2 lines
2.53.0
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?
Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Sorry I still don't get you.
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Quoted 85 lines
Thanks
Quoted 82 lines
Quoted 81 lines
Quoted 76 lines
drivers/vdpa/vdpa_user/vduse_dev.c +7
-0+spin_lock (&dev->msg_lock); +vduse_dev_broken (dev); +spin_unlock (&dev->msg_lock); + } }static bool vduse_vdpa_get_vq_ready (struct vdpa_device *vdpa,u16 idx)@@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config, dev->device_features = config->features; dev->device_id = config->device_id; dev->vendor_id = config->vendor_id; + dev->vduse_features = config->vduse_features; +dev_dbg (vduse_ctrl_dev,"Creating device %s with features 0x%llx" , + config->name, config->vduse_features); dev->nas = (dev->api_version <VDUSE_API_VERSION_1 ) ?1 : config->nas; dev->as =kcalloc (dev->nas,sizeof (dev->as[0 ]),GFP_KERNEL );include/uapi/linux/vduse.h +18
-1@@ -14,6 +14,9 @@#define VDUSE_API_VERSION_1 1 +/* The VDUSE instance expects a request for vq ready */ +#define VDUSE_F_QUEUE_READY 0 +/* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension. @@ -330,6 +333,7 @@ enum vduse_req_type {VDUSE_SET_STATUS ,VDUSE_UPDATE_IOTLB ,VDUSE_SET_VQ_GROUP_ASID , + VDUSE_SET_VQ_READY, };/** @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {__u32 padding; }; +/** +* struct vduse_vq_ready - Virtqueue ready request message +* @num: Virtqueue number +*/ +struct vduse_vq_ready { +__u32 num; +__u32 ready; +}; +/** * struct vduse_dev_request - control request * @type: request type @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {* @iova: IOVA range for updating * @iova_v2: IOVA range for updating if API_VERSION >= 1 * @vq_group_asid: ASID of a virtqueue group +* @vq_ready: Virtqueue ready request * @padding: padding * * Structure used by read(2) on /dev/vduse/$NAME. @@ -404,6 +418,10 @@ struct vduse_dev_request {*/ struct vduse_iova_range_v2 iova_v2;struct vduse_vq_group_asid vq_group_asid; + +/* Only if VDUSE_F_QUEUE_READY is negotiated */ +struct vduse_vq_ready vq_ready; +__u32 padding[32 ]; }; }; --Quoted 2 lines
2.53.0
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 120 lines · Eugenio Perez Martin
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Quoted 8 lines
Sorry I still don't get you.
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Friendly ping.
Michael S. Tsirkin
maintainerRe: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 124 lines · Eugenio Perez Martin
Quoted 122 lines · Eugenio Perez Martin
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Sorry I still don't get you.
Quoted 8 lines
Quoted 6 lines
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Friendly ping.
Jason, more comments? If this is to go in it has to go into linux next.
Signature · 1 line
MST
Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 133 lines · Michael S. Tsirkin
Quoted 131 lines · Eugenio Perez Martin
Quoted 128 lines · Eugenio Perez Martin
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Sorry I still don't get you.
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Quoted 3 lines
Quoted 1 line
Friendly ping.
Jason, more comments? If this is to go in it has to go into linux next.
A typo, basically I meant that reset is more heavyweight than queue ready. If we decide to check the response for queue ready, we need to check the reset as well.
But I'm fine if you think we can start from this.
Thanks
Quoted 4 lines
--
MST
Eugenio Perez Martin
maintainerRe: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 137 lines · Jason Wang
Quoted 135 lines · Michael S. Tsirkin
Quoted 132 lines · Eugenio Perez Martin
Quoted 120 lines · Eugenio Perez Martin
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Quoted 8 lines
Sorry I still don't get you.
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Friendly ping.
Quoted 8 lines
Quoted 1 line
Jason, more comments? If this is to go in it has to go into linux next.
A typo, basically I meant that reset is more heavyweight than queue ready. If we decide to check the response for queue ready, we need to check the reset as well.
But I'm fine if you think we can start from this.
If you mean something like this:
(quoted diff) +0 -0
b/drivers/vdpa/vdpa_user/vduse_dev.c
index 6202f6902fcd..23b850023417 100644
drivers/vdpa/vdpa_user/vduse_dev.c +2 -1
@@ -816,7 +816,8 @@ static int vduse_vdpa_reset(struct vdpa_device *vdpa)struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret =vduse_dev_set_status (dev,0 ); -vduse_dev_reset (dev); +if (ret ==0 ) +vduse_dev_reset (dev);return ret; }
---
I totally agree. But in my opinion, it's outside this series' scope. I can send a patch on top.
Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 155 lines · Eugenio Perez Martin
Quoted 137 lines · Jason Wang
Quoted 135 lines · Michael S. Tsirkin
Quoted 132 lines · Eugenio Perez Martin
Quoted 120 lines · Eugenio Perez Martin
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Quoted 8 lines
Sorry I still don't get you.
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Friendly ping.
Quoted 8 lines
Quoted 1 line
Jason, more comments? If this is to go in it has to go into linux next.
A typo, basically I meant that reset is more heavyweight than queue ready. If we decide to check the response for queue ready, we need to check the reset as well.
But I'm fine if you think we can start from this.
If you mean something like this:
drivers/vdpa/vdpa_user/vduse_dev.c +0
-0
drivers/vdpa/vdpa_user/vduse_dev.c static int vduse_vdpa_reset(struct vdpa_device *vdpa) · line 820
Quoted 6 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2
-1@@ -816,7 +816,8 @@ static int vduse_vdpa_reset(struct vdpa_device *vdpa)struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret =vduse_dev_set_status (dev,0 ); -vduse_dev_reset (dev); +if (ret ==0 ) +vduse_dev_reset (dev);
Something like this but we need to check if it's a timeout as queue ready.
Quoted 9 lines
drivers/vdpa/vdpa_user/vduse_dev.c +0
-1return ret; } ---I totally agree. But in my opinion, it's outside this series' scope. I can send a patch on top.
Works for me.
Thanks
Eugenio Perez Martin
maintainerRe: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature
Quoted 161 lines · Jason Wang
Quoted 158 lines · Eugenio Perez Martin
Quoted 137 lines · Jason Wang
Quoted 135 lines · Michael S. Tsirkin
Quoted 132 lines · Eugenio Perez Martin
Quoted 120 lines · Eugenio Perez Martin
Quoted 112 lines · Jason Wang
Quoted 102 lines · Eugenio Perez Martin
Quoted 91 lines · Jason Wang
Quoted 62 lines · Eugenio Pérez
Add the
VDUSE_F_QUEUE_READYfeature flag. This allows the kernel module to explicitly signal userspace when a specific virtqueue has been enabled.In scenarios like Live Migration of VirtIO net devices, the dataplane starts after the control virtqueue allowing QEMU to apply configuration in the destination device.
Signed-off-by Eugenio Pérez <[email protected]>---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Setvq_readybefore sending the message to the VDUSE userland instance, avoiding the need for SMP sync after receiving the message.--- drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++- include/uapi/linux/vduse.h | 18 ++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)drivers/vdpa/vdpa_user/vduse_dev.c +10
-1@@ -9,6 +9,7 @@*/ #include "linux/virtio_net.h" +#include <linux/bits.h> #include <linux/cleanup.h> #include <linux/init.h> #include <linux/module.h> @@ -53,7 +54,7 @@#define IRQ_UNBOUND -1/* Supported VDUSE features */ -static const uint64_t vduse_features; +static const uint64_t vduse_features =BIT_U64 (VDUSE_F_QUEUE_READY );/* * VDUSE instance have not asked the vduse API version, so assume 0. @@ -120,6 +121,7 @@ struct vduse_dev {char *name;struct mutex lock;spinlock_t msg_lock; +u64 vduse_features;u64 msg_unique;u32 msg_timeout;wait_queue_head_t waitq;@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, {struct vduse_dev *dev =vdpa_to_vduse (vdpa);struct vduse_virtqueue *vq = dev->vqs[idx]; +struct vduse_dev_msg msg = {0 }; +int r; vq->ready = ready; + +if (!(dev->vduse_features &BIT_U64 (VDUSE_F_QUEUE_READY ))) +return ; + + msg.req.type =VDUSE_SET_VQ_READY ;drivers/vdpa/vdpa_user/vduse_dev.c static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa, · line 624
Quoted 10 lines
drivers/vdpa/vdpa_user/vduse_dev.c +10
-0+ msg.req.vq_ready.num = idx; + msg.req.vq_ready.ready = !!ready; + + r =vduse_dev_msg_sync (dev, &msg); + +if (r <0 ) { +dev_dbg (&vdpa->dev,"device refuses to set vq %u ready %u" , + idx, ready); + +/* We can't do better than break the device in this case */ It's better to explain why we can't depend on
vduse_dev_msg_sync()here.For example it did:
if (unlikely(dev->broken)) return -EIO; init_waitqueue_head(&msg->waitq); spin_lock(&dev->msg_lock); if (unlikely(dev->broken)) { spin_unlock(&dev->msg_lock); return -EIO; }and
Quoted 8 lines
if (!msg->completed) { list_del(&msg->list); msg->resp.result = VDUSE_REQ_RESULT_FAILED; /* Mark the device as malfunction when there is a timeout */ if (!ret) vduse_dev_broken(dev); }Quoted 6 lines
I'm not sure I follow you here.
We can't do better than breaking the device because the function returns no error state, and the caller does not expect an error code. Do you mean we can't depend on
vduse_dev_msg_syncto callvduse_dev_broken(dev)by itself?Quoted 5 lines
I think I meant, reset seems to be more heavyweight than suspend.
So if reset can fail, I don't see reason ot break device only for suspend failure.
Quoted 8 lines
Sorry I still don't get you.
This series does not implement suspend at all. It doesn't modify the VDUSE device reset or the virtio reset behavior. It only implements the vq ready message for the device. If the device returns an error from that operation, what is your proposal for when the driver sends new messages like resume?
Friendly ping.
Quoted 8 lines
Quoted 1 line
Jason, more comments? If this is to go in it has to go into linux next.
A typo, basically I meant that reset is more heavyweight than queue ready. If we decide to check the response for queue ready, we need to check the reset as well.
But I'm fine if you think we can start from this.
If you mean something like this:
drivers/vdpa/vdpa_user/vduse_dev.c +0
-0@@ -816,7 +816,8 @@ static int vduse_vdpa_reset(struct vdpa_device *vdpa)struct vduse_dev *dev =vdpa_to_vduse (vdpa);int ret =vduse_dev_set_status (dev,0 );
Quoted 6 lines
drivers/vdpa/vdpa_user/vduse_dev.c static int vduse_vdpa_reset(struct vdpa_device *vdpa) · line 820
Quoted 3 lines
drivers/vdpa/vdpa_user/vduse_dev.c +2
-1-vduse_dev_reset (dev); +if (ret ==0 ) +vduse_dev_reset (dev);Something like this but we need to check if it's a timeout as queue ready.
What should VDUSE or vdpa layer do differently if it is a timeout vs returning an error?
Quoted 14 lines
Quoted 9 lines
drivers/vdpa/vdpa_user/vduse_dev.c +0
-1return ret; } ---I totally agree. But in my opinion, it's outside this series' scope. I can send a patch on top.
Works for me.
Thanks