vdpa: reject F_ENABLE_AFTER_DRIVER_OK if backend does not support it
Subsystems (5)
Eugenio Pérez
maintainerPATCH 1/1 vdpa: reject F_ENABLE_AFTER_DRIVER_OK if backend does not support it
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.
[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/vhost/vdpa.c +5 -2
@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v))
Signature · 1 line
2.39.3
Michael S. Tsirkin
maintainerQuoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.
Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>
Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want RING_ACCESS_AFTER_KICK or some such.
Quoted 39 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-2@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v))Signature · 1 line
2.39.3
Eugenio Perez Martin
maintainerQuoted 12 lines · Michael S. Tsirkin
Quoted 10 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Quoted 7 lines
Quoted 2 lines
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.
Current devices do not support that semantic. My plan was to convert it in vp_vdpa if needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.
The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* Offer VHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, at vdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.
But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after DRIVER_OK? That way no new feature flag is needed.
Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Michael S. Tsirkin
maintainerQuoted 16 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 12 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>
Quoted 6 lines
Quoted 4 lines
Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Current devices do not support that semantic.
Which devices specifically access the ring after DRIVER_OK but before a kick?
Quoted 62 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Eugenio Perez Martin
maintainerQuoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.
Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after driver_ok, even without a kick, was considered work of the parent driver.
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Quoted 63 lines
Quoted 62 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Michael S. Tsirkin
maintainerQuoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for ENABLE_AFTER_DRIVER_OK too, no?
Quoted 63 lines
Quoted 63 lines
Quoted 62 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.
Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?
My understanding is [1] assuming ACCESS_AFTER_KICK. This seems sub-optimal than assuming ENABLE_AFTER_DRIVER_OK.
But this reminds me one thing, as the thread is going too long, I wonder if we simply assume ENABLE_AFTER_DRIVER_OK if RING_RESET is supported?
Thanks
Quoted 67 lines
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Michael S. Tsirkin
maintainerQuoted 48 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?
Quoted 8 lines
My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Thanks
I don't see what does one have to do with another ...
I think with RING_RESET we had another solution, enable rings mapping them to a zero page, then reset and re-enable later.
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?
Quoted 6 lines
Quoted 1 line
Thanks
I don't see what does one have to do with another ...
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.
As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVA
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Eugenio Perez Martin
maintainerQuoted 63 lines · Jason Wang
Quoted 61 lines · Michael S. Tsirkin
Quoted 48 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?Quoted 8 lines
My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Thanks
I don't see what does one have to do with another ...
Quoted 8 lines
Quoted 2 lines
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVA
I think we're not on the same page here.
As I understood, rings mapped to a zero page means essentially an avail ring whose avail_idx is always 0, offered to the device instead of the guest's ring. Once all CVQ commands are processed, we use RING_RESET to switch to the right ring, being guest's or SVQ vring.
Quoted 72 lines
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 74 lines · Eugenio Perez Martin
Quoted 63 lines · Jason Wang
Quoted 61 lines · Michael S. Tsirkin
Quoted 48 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?Quoted 8 lines
My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Thanks
I don't see what does one have to do with another ...
Quoted 8 lines
Quoted 2 lines
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVA
Quoted 6 lines
I think we're not on the same page here.
As I understood, rings mapped to a zero page means essentially an avail ring whose
avail_idxis always 0, offered to the device instead of the guest's ring. Once all CVQ commands are processed, we useRING_RESETto switch to the right ring, being guest's or SVQ vring.
I get this. This seems more complicated in the destination: shadow vq + ASID?
Thanks
Quoted 76 lines
Quoted 75 lines
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 76 lines · Jason Wang
Quoted 74 lines · Eugenio Perez Martin
Quoted 71 lines · Jason Wang
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Quoted 6 lines
Quoted 1 line
Thanks
I don't see what does one have to do with another ...
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVA
Quoted 8 lines
Quoted 6 lines
I think we're not on the same page here.
As I understood, rings mapped to a zero page means essentially an avail ring whose
avail_idxis always 0, offered to the device instead of the guest's ring. Once all CVQ commands are processed, we useRING_RESETto switch to the right ring, being guest's or SVQ vring.I get this. This seems more complicated in the destination: shadow vq + ASID?
So it's something like:
1) set all vq ASID to shadow virtqueue
2) do not add any bufs to data qp (stick 0 as avail index)
3) start to restore states via cvq
4) ring_rest for dataqp
5) set_vq_state for dataqp
6) re-initialize dataqp address etc
7) set data QP ASID to guest
8) set queue_enable
?
Thanks
Quoted 79 lines
Thanks
Quoted 76 lines
Quoted 75 lines
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Eugenio Perez Martin
maintainerQuoted 98 lines · Jason Wang
Quoted 76 lines · Jason Wang
Quoted 74 lines · Eugenio Perez Martin
Quoted 71 lines · Jason Wang
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Quoted 6 lines
Quoted 1 line
Thanks
I don't see what does one have to do with another ...
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVAQuoted 8 lines
Quoted 6 lines
I think we're not on the same page here.
As I understood, rings mapped to a zero page means essentially an avail ring whose
avail_idxis always 0, offered to the device instead of the guest's ring. Once all CVQ commands are processed, we useRING_RESETto switch to the right ring, being guest's or SVQ vring.I get this. This seems more complicated in the destination: shadow vq + ASID?
So it's something like:
1) set all vq ASID to shadow virtqueue
2) do not add any bufs to data qp (stick 0 as avail index)
3) start to restore states via cvq
4)ring_restfor dataqp
5)set_vq_statefor dataqp
6) re-initialize dataqp address etc
7) set data QP ASID to guest
8) setqueue_enable
Quoted 2 lines
?
I think the change of ASID is not needed, as the guest cannot access the device in that timeframe anyway. Moreover, it may require HW support. So steps 1 and 7 are not needed.
Apart from that, the process is right.
Quoted 82 lines
Thanks
Quoted 79 lines
Thanks
Quoted 76 lines
Quoted 75 lines
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 100 lines · Eugenio Perez Martin
Quoted 98 lines · Jason Wang
Quoted 76 lines · Jason Wang
Quoted 74 lines · Eugenio Perez Martin
Quoted 71 lines · Jason Wang
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 12 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been
offered to it with VHOST_GET_BACKEND_FEATURES, the current code will not
complain for it.Since there is no specific reason for any parent to reject that backend
feature bit when it has been proposed, let's control it at vdpa frontend
level. Future patches may move this control to the parent driver.Fixes: 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")
Signed-off-by: Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Quoted 6 lines
Quoted 1 line
Thanks
I don't see what does one have to do with another ...
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVAQuoted 8 lines
Quoted 6 lines
I think we're not on the same page here.
As I understood, rings mapped to a zero page means essentially an avail ring whose
avail_idxis always 0, offered to the device instead of the guest's ring. Once all CVQ commands are processed, we useRING_RESETto switch to the right ring, being guest's or SVQ vring.I get this. This seems more complicated in the destination: shadow vq + ASID?
So it's something like:
1) set all vq ASID to shadow virtqueue
2) do not add any bufs to data qp (stick 0 as avail index)
3) start to restore states via cvq
4)ring_restfor dataqp
5)set_vq_statefor dataqp
6) re-initialize dataqp address etc
7) set data QP ASID to guest
8) setqueue_enable
Quoted 5 lines
Quoted 2 lines
?
I think the change of ASID is not needed, as the guest cannot access the device in that timeframe anyway.
Yes but after the restore, we still want to shadow cvq, so ASID is still needed?
Thanks
Quoted 89 lines
Moreover, it may require HW
support. So steps 1 and 7 are not needed.Apart from that, the process is right.
Quoted 82 lines
Thanks
Quoted 79 lines
Thanks
Quoted 76 lines
Quoted 75 lines
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst
commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
---
drivers/vhost/vdpa.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index e1abf29fed5b..a7e554352351 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
{
struct vhost_vdpa *v = filep->private_data;
struct vhost_dev *d = &v->vdev;
+ const struct vdpa_config_ops *ops = v->vdpa->config;
void __user *argp = (void __user *)arg;
u64 __user *featurep = argp;
- u64 features;
+ u64 features, parent_features = 0;
long r = 0;if (cmd == VHOST_SET_BACKEND_FEATURES) {
if (copy_from_user(&features, featurep, sizeof(features)))
return -EFAULT;
+ if (ops->get_backend_features)
+ parent_features = ops->get_backend_features(v->vdpa);
if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
BIT_ULL(VHOST_BACKEND_F_SUSPEND) |
BIT_ULL(VHOST_BACKEND_F_RESUME) |
- BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK)))
+ parent_features))
return -EOPNOTSUPP;
if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) &&
!vhost_vdpa_can_suspend(v))
--
2.39.3
Eugenio Perez Martin
maintainerQuoted 102 lines · Jason Wang
Quoted 100 lines · Eugenio Perez Martin
Quoted 98 lines · Jason Wang
Quoted 76 lines · Jason Wang
Quoted 74 lines · Eugenio Perez Martin
Quoted 71 lines · Jason Wang
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
On Mon, Jul 03, 2023 at 04:22:18PM +0200, Eugenio Pérez wrote:
With the current code it is accepted as long as userland send it.Although userland should not set a feature flag that has not been
offered to it with VHOST_GET_BACKEND_FEATURES, the current code will not
complain for it.Since there is no specific reason for any parent to reject that backend
feature bit when it has been proposed, let's control it at vdpa frontend
level. Future patches may move this control to the parent driver.Fixes: 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")
Signed-off-by: Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack
upstream at all, I merged it in next just to give it some testing.
We want RING_ACCESS_AFTER_KICK or some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.Quoted 8 lines
Quoted 6 lines
I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?Quoted 6 lines
Quoted 1 line
Thanks
I don't see what does one have to do with another ...
I think with
RING_RESETwe had another solution, enable rings mapping them to a zero page, then reset and re-enable later.As discussed before, this seems to have some problems:
1) The behaviour is not clarified in the document
2) zero is a valid IOVAQuoted 8 lines
Quoted 6 lines
I think we're not on the same page here.
As I understood, rings mapped to a zero page means essentially an avail ring whose
avail_idxis always 0, offered to the device instead of the guest's ring. Once all CVQ commands are processed, we useRING_RESETto switch to the right ring, being guest's or SVQ vring.I get this. This seems more complicated in the destination: shadow vq + ASID?
So it's something like:
1) set all vq ASID to shadow virtqueue
2) do not add any bufs to data qp (stick 0 as avail index)
3) start to restore states via cvq
4)ring_restfor dataqp
5)set_vq_statefor dataqp
6) re-initialize dataqp address etc
7) set data QP ASID to guest
8) setqueue_enable
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
?
I think the change of ASID is not needed, as the guest cannot access the device in that timeframe anyway.
Yes but after the restore, we still want to shadow cvq, so ASID is still needed?
Device or parent driver support for ASID is needed to shadow only CVQ. Bue the device may not support the switch of ASID after DRIVER_OK.
Since dataplane can go in passthrough ASID all the time, we don't need to switch it after DRIVER_OK.
Quoted 92 lines
Thanks
Quoted 89 lines
Moreover, it may require HW
support. So steps 1 and 7 are not needed.Apart from that, the process is right.
Quoted 82 lines
Thanks
Quoted 79 lines
Thanks
Quoted 76 lines
Quoted 75 lines
Thanks
Quoted 69 lines
Quoted 68 lines
Quoted 68 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst
commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
---
drivers/vhost/vdpa.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index e1abf29fed5b..a7e554352351 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
{
struct vhost_vdpa *v = filep->private_data;
struct vhost_dev *d = &v->vdev;
+ const struct vdpa_config_ops *ops = v->vdpa->config;
void __user *argp = (void __user *)arg;
u64 __user *featurep = argp;
- u64 features;
+ u64 features, parent_features = 0;
long r = 0;if (cmd == VHOST_SET_BACKEND_FEATURES) {
if (copy_from_user(&features, featurep, sizeof(features)))
return -EFAULT;
+ if (ops->get_backend_features)
+ parent_features = ops->get_backend_features(v->vdpa);
if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
BIT_ULL(VHOST_BACKEND_F_SUSPEND) |
BIT_ULL(VHOST_BACKEND_F_RESUME) |
- BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK)))
+ parent_features))
return -EOPNOTSUPP;
if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) &&
!vhost_vdpa_can_suspend(v))
--
2.39.3
Eugenio Perez Martin
maintainerQuoted 47 lines · Jason Wang
Quoted 45 lines · Michael S. Tsirkin
Quoted 42 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 5 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Quoted 9 lines
Quoted 1 line
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?
The problem with that is that the device needs to support all RING_RESET, like to be able to change vq address etc after DRIVER_OK. Not all HW support it.
We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 70 lines
Thanks
Quoted 67 lines
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 30 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?
The PDS vdpa device can deal with a call to .set_vq_ready after DRIVER_OK is set. And I'm told that our VQ activity should start without a kick.
Our vdpa device FW doesn't currently have support for VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.
sln
Quoted 106 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Quoted 9 lines
Quoted 1 line
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 70 lines
Thanks
Quoted 67 lines
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Michael S. Tsirkin
maintainerQuoted 36 lines · Shannon Nelson
Quoted 30 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?
Quoted 8 lines
The PDS vdpa device can deal with a call to .set_vq_ready after
DRIVER_OKis set. And I'm told that our VQ activity should start without a kick.Our vdpa device FW doesn't currently have support for
VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.sln
OK it seems clear at least in the current version pds needs VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
However can we also code up the RING_RESET path as the default? Then down the road vendors can choose what to do.
Quoted 98 lines
Quoted 98 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Quoted 9 lines
Quoted 1 line
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 63 lines
Thanks
Quoted 61 lines
Quoted 61 lines
Quoted 61 lines
Quoted 61 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 37 lines
Quoted 37 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-2@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ; if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) &&
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
drivers/vhost/vdpa.c +0
-1!vhost_vdpa_can_suspend(v)) --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerQuoted 44 lines · Michael S. Tsirkin
Quoted 42 lines · Shannon Nelson
Quoted 30 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?The PDS vdpa device can deal with a call to .set_vq_ready after
DRIVER_OKis set. And I'm told that our VQ activity should start without a kick.Our vdpa device FW doesn't currently have support for
VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.
Quoted 8 lines
Quoted 2 lines
sln
OK it seems clear at least in the current version pds needs
VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
However can we also code up theRING_RESETpath as the default? Then down the road vendors can choose what to do.
Yes, the RING_RESET path can be coded & tested for vp_vdpa, for example. I'm ok with making it the default, and let _F_ENABLE_AFTER_DRIVER_OK as a fallback.
Quoted 111 lines
Quoted 110 lines
Quoted 110 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 5 lines
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Quoted 9 lines
Quoted 1 line
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 70 lines
Thanks
Quoted 67 lines
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Michael S. Tsirkin
maintainerQuoted 54 lines · Eugenio Perez Martin
Quoted 44 lines · Michael S. Tsirkin
Quoted 42 lines · Shannon Nelson
Quoted 30 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?The PDS vdpa device can deal with a call to .set_vq_ready after
DRIVER_OKis set. And I'm told that our VQ activity should start without a kick.Our vdpa device FW doesn't currently have support for
VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.Quoted 8 lines
Quoted 2 lines
sln
OK it seems clear at least in the current version pds needs
VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
However can we also code up theRING_RESETpath as the default? Then down the road vendors can choose what to do.
Quoted 3 lines
Yes, the
RING_RESETpath can be coded & tested forvp_vdpa, for example. I'm ok with making it the default, and let_F_ENABLE_AFTER_DRIVER_OKas a fallback.
Sounds good.
Quoted 111 lines
Quoted 111 lines
Quoted 110 lines
Quoted 110 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 5 lines
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
[1] https://lists.nongnu.org/archive/html/qemu-devel/2023-01/msg02775.html
Quoted 9 lines
Quoted 1 line
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 70 lines
Thanks
Quoted 67 lines
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Shannon Nelson
Quoted 19 lines · Eugenio Perez Martin
Quoted 18 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Eugenio Perez Martin
Quoted 13 lines · Michael S. Tsirkin
Quoted 12 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?The PDS vdpa device can deal with a call to .set_vq_ready after
DRIVER_OKis set. And I'm told that our VQ activity should start without a kick.
Quoted 8 lines
Quoted 5 lines
Our vdpa device FW doesn't currently have support for
VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.sln
OK it seems clear at least in the current version pds needs
VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
However can we also code up theRING_RESETpath as the default?
What's the rationale of making RING_RESET path the default? Noted this is on a performance critical path (for live migration downtime), did we ever get consensus from every or most hardware vendors that RING_RESET has lower cost in terms of latency overall than ENABLE_AFTER_DRIVER_OK? I think (RING)RESET in general falls on the slow path for hardware, while I assume either RING_RESET or ENABLE_AFTER_DRIVER_OK doesn't matters much on software backed vdpa e.g. vp_vdpa. Maybe should make ENABLE_AFTER_DRIVER_OK as the default?
-Siwei
Quoted 104 lines
Then down the road vendors can choose what to do.
Quoted 79 lines
Quoted 79 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
Quoted 9 lines
Quoted 2 lines
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 49 lines
Thanks
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 24 lines
Quoted 24 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +2
-1@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/vhost/vdpa.c +2
-0if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa); if (features & ~(VHOST_VDPA_BACKEND_FEATURES | BIT_ULL(VHOST_BACKEND_F_SUSPEND) |Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1
-2BIT_ULL (VHOST_BACKEND_F_RESUME ) | - BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) + parent_features)) return -EOPNOTSUPP;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 50 lines · Si-Wei Liu
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Shannon Nelson
Quoted 19 lines · Eugenio Perez Martin
Quoted 18 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Eugenio Perez Martin
Quoted 13 lines · Michael S. Tsirkin
Quoted 12 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?The PDS vdpa device can deal with a call to .set_vq_ready after
DRIVER_OKis set. And I'm told that our VQ activity should start without a kick.Quoted 8 lines
Quoted 5 lines
Our vdpa device FW doesn't currently have support for
VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.sln
OK it seems clear at least in the current version pds needs
VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
However can we also code up theRING_RESETpath as the default?What's the rationale of making
RING_RESETpath the default? Noted this is on a performance critical path (for live migration downtime), did we ever get consensus from every or most hardware vendors thatRING_RESEThas lower cost in terms of latency overall thanENABLE_AFTER_DRIVER_OK? I think (RING)RESET in general falls on the slow path for hardware, while I assume eitherRING_RESETorENABLE_AFTER_DRIVER_OKdoesn't matters much on software backed vdpa e.g.vp_vdpa. Maybe should makeENABLE_AFTER_DRIVER_OKas the default?
Quoted 1 line
-Siwei
Coming from the spec RING_RESET has clearer semantics. As long as we support it it is not critical which one is the default though.
Quoted 96 lines
Quoted 96 lines
Then down the road vendors can choose what to do.
Quoted 90 lines
Quoted 90 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
Quoted 9 lines
Quoted 2 lines
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 60 lines
Thanks
Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 35 lines
Quoted 35 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-2@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vhost/vdpa.c +0
-1if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 50 lines · Michael S. Tsirkin
Quoted 49 lines · Si-Wei Liu
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Shannon Nelson
Quoted 19 lines · Eugenio Perez Martin
Quoted 18 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Eugenio Perez Martin
Quoted 13 lines · Michael S. Tsirkin
Quoted 12 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?The PDS vdpa device can deal with a call to .set_vq_ready after
DRIVER_OKis set. And I'm told that our VQ activity should start without a kick.Quoted 8 lines
Quoted 5 lines
Our vdpa device FW doesn't currently have support for
VIRTIO_F_RING_RESET, but I believe it could be added without too much trouble.sln
OK it seems clear at least in the current version pds needs
VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
However can we also code up theRING_RESETpath as the default?What's the rationale of making
RING_RESETpath the default? Noted this is on a performance critical path (for live migration downtime), did we ever get consensus from every or most hardware vendors thatRING_RESEThas lower cost in terms of latency overall thanENABLE_AFTER_DRIVER_OK? I think (RING)RESET in general falls on the slow path for hardware, while I assume eitherRING_RESETorENABLE_AFTER_DRIVER_OKdoesn't matters much on software backed vdpa e.g.vp_vdpa. Maybe should makeENABLE_AFTER_DRIVER_OKas the default?
Quoted 2 lines
Quoted 1 line
-Siwei
Coming from the spec
RING_RESEThas clearer semantics.
Spec doesn't have clearer semantics on vdpa specifics - such as how does RING_RESET interoperate with ASID?
Quoted 2 lines
As long as we support it it is not critical which one is the default though.
The point is vdpa vendor drivers may implement RING_RESET for a different purpose than live migration. In case they support both I don't see a reason why it has to fallback to a slower path given there's a faster path. May we should leave this to vendor driver to decide, but I am not sure.
-Siwei
Quoted 105 lines
Quoted 105 lines
Quoted 105 lines
Then down the road vendors can choose what to do.
Quoted 78 lines
Quoted 78 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Previous versions of the QEMU LM series did a spurious kick to start traffic at the LM destination [1]. When it was proposed, that spurious kick was removed from the series because to check for descriptors after
driver_ok, even without a kick, was considered work of the parent driver.I'm ok to go back to this spurious kick, but I'm not sure if the hw will read the ring before the kick actually. I can ask.
Thanks!
Quoted 9 lines
Quoted 2 lines
Let's find out. We need to check for
ENABLE_AFTER_DRIVER_OKtoo, no?My understanding is [1] assuming
ACCESS_AFTER_KICK. This seems sub-optimal than assumingENABLE_AFTER_DRIVER_OK.But this reminds me one thing, as the thread is going too long, I wonder if we simply assume
ENABLE_AFTER_DRIVER_OKifRING_RESETis supported?The problem with that is that the device needs to support all
RING_RESET, like to be able to change vq address etc afterDRIVER_OK. Not all HW support it.We just need the subset of having the dataplane freezed until all CVQ commands have been consumed. I'm sure current vDPA code already supports it in some devices, like MLX and PSD.
Thanks!
Quoted 48 lines
Thanks
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 24 lines
Quoted 24 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +2
-1@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/vhost/vdpa.c +2
-0if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa); if (features & ~(VHOST_VDPA_BACKEND_FEATURES | BIT_ULL(VHOST_BACKEND_F_SUSPEND) |Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1
-2BIT_ULL (VHOST_BACKEND_F_RESUME ) | - BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) + parent_features)) return -EOPNOTSUPP;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.
Quoted 4 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?
Vhost-net is one example at last. It polls a socket as well, so it starts to access the ring immediately after DRIVER_OK.
Thanks
Quoted 47 lines
Quoted 47 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 25 lines
Quoted 25 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +2
-1@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vhost/vdpa.c +2
-0if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa); if (features & ~(VHOST_VDPA_BACKEND_FEATURES | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) |Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
drivers/vhost/vdpa.c +1
-2- BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 26 lines · Jason Wang
Quoted 25 lines · Michael S. Tsirkin
Quoted 22 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Quoted 7 lines
Quoted 2 lines
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Current devices do not support that semantic.
Quoted 7 lines
Quoted 2 lines
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Vhost-net is one example at last. It polls a socket as well, so it starts to access the ring immediately after
DRIVER_OK.Thanks
For sure but that is not vdpa.
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Jason Wang
Quoted 27 lines · Michael S. Tsirkin
Quoted 16 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 12 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Quoted 6 lines
Quoted 4 lines
Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?
Quoted 7 lines
Quoted 4 lines
Vhost-net is one example at last. It polls a socket as well, so it starts to access the ring immediately after
DRIVER_OK.Thanks
For sure but that is not vdpa.
Somehow via vp_vdpa that I'm usually testing vDPA patches.
The problem is that it's very hard to audit all vDPA devices now if it is not defined in the spec before they are invented.
Thanks
Quoted 57 lines
Quoted 57 lines
Quoted 57 lines
Quoted 57 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 34 lines
Quoted 34 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +4
-2@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa); if (features & ~(VHOST_VDPA_BACKEND_FEATURES | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK )))
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
drivers/vhost/vdpa.c +1
-1+ parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 38 lines · Jason Wang
Quoted 37 lines · Michael S. Tsirkin
Quoted 33 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Eugenio Perez Martin
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez
With the current code it is accepted as long as userland send it.
Although userland should not set a feature flag that has not been offered to it with
VHOST_GET_BACKEND_FEATURES, the current code will not complain for it.Quoted 6 lines
Since there is no specific reason for any parent to reject that backend feature bit when it has been proposed, let's control it at vdpa frontend level. Future patches may move this control to the parent driver.
Fixes 967800d2d52e ("vdpa: accept VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK backend feature")Signed-off-by Eugenio Pérez <[email protected]>Please do send v3. And again, I don't want to send "after driver ok" hack upstream at all, I merged it in next just to give it some testing. We want
RING_ACCESS_AFTER_KICKor some such.Quoted 4 lines
Quoted 1 line
Current devices do not support that semantic.
Which devices specifically access the ring after
DRIVER_OKbut before a kick?Vhost-net is one example at last. It polls a socket as well, so it starts to access the ring immediately after
DRIVER_OK.Thanks
Quoted 8 lines
Quoted 1 line
For sure but that is not vdpa.
Somehow via
vp_vdpathat I'm usually testing vDPA patches.The problem is that it's very hard to audit all vDPA devices now if it is not defined in the spec before they are invented.
Thanks
vp_vdpa is exactly the part that bothers me. If on the host it is backed by e.g. vhost-user then it does not work. And you don't know what is backing it.
OTOH it supports RING_RESET ...
So, proposal: include both this solution and for drivers vp_vdpa the RING_RESET trick.
Hmm?
Quoted 66 lines
Quoted 66 lines
Quoted 65 lines
Quoted 65 lines
Quoted 64 lines
My plan was to convert
it invp_vdpaif needed, and reuse the current vdpa ops. Sorry if I was not explicit enough.The only solution I can see to that is to trap & emulate in the vdpa (parent?) driver, as talked in virtio-comment. But that complicates the architecture:
* OfferVHOST_BACKEND_F_RING_ACCESS_AFTER_KICK
* Store vq enable state separately, atvdpa->config->set_vq_ready(true), but not transmit that enable to hw
* Store the doorbell state separately, but do not configure it to the device directly.But how to recover if the device cannot configure them at kick time, for example?
Maybe we can just fail if the parent driver does not support enabling the vq after
DRIVER_OK? That way no new feature flag is needed.Thanks!
Quoted 41 lines
Quoted 40 lines
---
Sent with Fixes: tag pointing to git.kernel.org/pub/scm/linux/kernel/git/mst commit. Please let me know if I should send a v3 of [1] instead.[1] https://lore.kernel.org/lkml/[email protected]/T/
--- drivers/vhost/vdpa.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/vhost/vdpa.c +5
-3@@ -681,18 +681,21 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, {struct vhost_vdpa *v = filep->private_data;struct vhost_dev *d = &v->vdev; +const struct vdpa_config_ops *ops = v->vdpa->config;void __user *argp = (void __user *)arg;u64 __user *featurep = argp; -u64 features; +u64 features, parent_features =0 ;long r =0 ;if (cmd ==VHOST_SET_BACKEND_FEATURES ) {if (copy_from_user (&features, featurep,sizeof (features)))return -EFAULT ; +if (ops->get_backend_features) + parent_features = ops->get_backend_features (v->vdpa);if (features & ~(VHOST_VDPA_BACKEND_FEATURES |BIT_ULL (VHOST_BACKEND_F_SUSPEND ) |BIT_ULL (VHOST_BACKEND_F_RESUME ) | -BIT_ULL (VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK ))) + parent_features))return -EOPNOTSUPP ;if ((features &BIT_ULL (VHOST_BACKEND_F_SUSPEND )) && !vhost_vdpa_can_suspend (v)) --2.39.3